I need help about this specific scenario.
I can get this information from
using the key , I have to call this endpoint
At the end, using this sessionId and performing right POST actions , I can navigate inside all pages I need.
I’m able to simulate all steps using
scrapy shell with
regEx (and all work fine), but I don’t know how to manage these steps inside a scrapy spider before starting data extraction.
Can someone help me out?
You need to start with base URL
http://www.example.com/index.php by calling it in start request method and write its callback and extract information from other endpoint and take that result into other callback and then you can start scraping process.
You need to implement in the following way
def parse_authentication_token(self, response):
//extract token or whatever require and then call supers parse
yield from super().parse()
return Request(url, callback=self.parse_authentication_token)