How to handle errors inside yield for Scrapy?

Question

I’m using scrapy to yield in each product URL to get a JSON format and create my .csv which is working great.

    def parse_variants(self, response):
        text = '.js'
        url = response.xpath('//link[2]/@href').get()
        full_url = url + text
        parsing = requests.get(full_url).json()
        for x in range(0,len(parsing['variants'])):
            yield {
                'added_on_store': parsing['created_at'],
                'type': parsing['type'],
                'product_name': parsing['title'],
                'variant_name': parsing['variants'][x]['title'],
                'current_price': (parsing['variants'][x]['price']) // 100,
                'original_price': parsing['variants'][x]['compare_at_price'],
                'bar_code/GTIN': parsing['variants'][x]['barcode']
            }

By default the prices comes with 4 digits and I need to remove the last 2 (that’s why I use ‘ // 100’).
It works good for ‘current_price’ but not necessarly for ‘original_price’ since this value can be empty.

If ‘original_price’ is empty it will return an error which make sense since it can’t floor None by 100.

This is why I want to have an error handler or a default parameter if this is empty but still use ‘// 100’ if there is something.

Any suggestion?

Thank you.

Asked By: Yvan S

||

Source

Answer 1

The simplest method would be to use a conditional statement to set the value.

For example:

def parse_variants(self, response):
    text = '.js'
    url = response.xpath('//link[2]/@href').get()
    full_url = url + text
    parsing = requests.get(full_url).json()
    for x in range(0,len(parsing['variants'])):
        original_price = parsing['variants'][x]['compare_at_price']
        original_price = original_price // 100 if original_price else 0
        yield {
            'added_on_store': parsing['created_at'],
            'type': parsing['type'],
            'product_name': parsing['title'],
            'variant_name': parsing['variants'][x]['title'],
            'current_price': (parsing['variants'][x]['price']) // 100,
            'original_price': original_price,
            'bar_code/GTIN': parsing['variants'][x]['barcode']
        }

Answered By: Alexander

How to handle errors inside yield for Scrapy?

Question:

Answers: