yield scrapy.Request does not invoke parse function on each iteration

Question

in my code i have to functions inside scrapy class.
start_request takes data from excel workbook and assigns value to plate_num_xlsx variable.

def start_requests(self):
    df=pd.read_excel('data.xlsx')
    columnA_values=df['PLATE']
    for row in columnA_values:
        global  plate_num_xlsx
        plate_num_xlsx=row
        print("+",plate_num_xlsx)
        base_url =f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num_xlsx}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
        url=base_url
        yield scrapy.Request(url,callback=self.parse)

But, on each iteration it should invoke parse() method of Scrapy class and inside that function with each iterated newly value of plate_num_xlsx needs to compare the parsed value,
As I understood after print statements it first takes all values, assigns them then only with last value assigned calls parse() method. But for my crawler to function properly I need each time assigning happens to call and use that value inside def parse().
code is below;

import scrapy
from scrapy.crawler import CrawlerProcess
import pandas as pd

itemList=[]
class plateScraper(scrapy.Spider):
    name = 'scrapePlate'
    allowed_domains = ['dvlaregistrations.dvla.gov.uk']

    def start_requests(self):
        df=pd.read_excel('data.xlsx')
        columnA_values=df['PLATE']
        for row in columnA_values:
            global  plate_num_xlsx
            plate_num_xlsx=row
            print("+",plate_num_xlsx)
            base_url =f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num_xlsx}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
            url=base_url
            yield scrapy.Request(url,callback=self.parse)

    def parse(self, response):

        for row in response.css('div.resultsstrip'):
            plate = row.css('a::text').get()
            price = row.css('p::text').get()
            a = plate.replace(" ", "").strip()
            print(plate_num_xlsx,a,a == plate_num_xlsx)
            if plate_num_xlsx==plate.replace(" ","").strip():
                item= {"plate": plate.strip(), "price": price.strip()}
                itemList.append(item)
                yield  item
            else:
                item = {"plate": plate_num_xlsx, "price": "-"}
                itemList.append(item)
                yield item

        with pd.ExcelWriter('output_res.xlsx', mode='r+',if_sheet_exists='overlay') as writer:
            df_output = pd.DataFrame(itemList)
            df_output.to_excel(writer, sheet_name='result', index=False, header=True)

process = CrawlerProcess()
process.crawl(plateScraper)
process.start()

Asked By: xlmaster

||

Source

Answer 1

Using global variables with scrapy in this manner will not work due to it’s asynchronous runtime behavior. What you can do alternatively is pass the plate_num_xlsx variable as a callback keyword argument to the request object itself.

for example:

            plate_num_xlsx=row
            base_url =f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num_xlsx}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
            url=base_url
            yield scrapy.Request(url,callback=self.parse, cb_kwargs={'plate_num_xlsx': plate_num_xlsx})



    def parse(self, response, plate_num_xlsx=None):
        for row in response.css('div.resultsstrip'):
            plate = row.css('a::text').get()
            price = row.css('p::text').get()
            ...

Now the variable will be included as a parameter to the parse function.

Answered By: Alexander

yield scrapy.Request does not invoke parse function on each iteration

Question:

Answers: