Scrapy – Retrieve Authentication Token from javascript script response


I need help about this specific scenario.


  1. Calling site

I can get this information from <script> tag

using the key , I have to call this endpoint

for retrieving the SessionID which is stored inside the javascript response

-- omitted


-- omitted

At the end, using this sessionId and performing right POST actions , I can navigate inside all pages I need.

My stalemate

I’m able to simulate all steps using scrapy shell with regEx (and all work fine), but I don’t know how to manage these steps inside a scrapy spider before starting data extraction.

Can someone help me out?

Asked By: MetalMad



You need to start with base URL by calling it in start request method and write its callback and extract information from other endpoint and take that result into other callback and then you can start scraping process.

You need to implement in the following way

class CrawlSpider(scrapy.CrawlSpider):

   def parse_authentication_token(self, response):
      //extract token or whatever require and then call supers parse
      yield from super().parse()

   def start_requests(self):
       return Request(url, callback=self.parse_authentication_token)
Answered By: Ahmed Buksh
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.