scrapy-splash

Scrapy : ValueError: XPath error: Invalid expression Question: I am trying to learn scrapy for a project. I receive this error ValueError: XPath error: Invalid expression but I don’t understand what is wrong in my script. It’s my script def parse(self, response): yield { ‘user_agent’: str(response.request.headers[‘User-Agent’]), ‘links’ : response.xpath(‘//a[@class="sc-996f251d-0 leAMGT"]/@href’).getall() } next = response.xpath(‘//a[@title="Page suivante"]/’) …

Unable to scrape items using scrapy [solved] Question: I am trying to webscrape the name, price, and description of products listed on an online shop. The website link is https://eshop.nomin.mn/n-foods.html When I look through the HTML code of the page, I get the relevant div class containers but when I reference it in my code …

Trying to fake and rotating user agents Question: I am trying to fake user agents as well as rotate them in Python. I found a tutorial online about how to do this with Scrapy using scrapy-useragents package. I scrape the webpage, https://www.whatsmyua.info/, in order to check my user agent to see if it is different …

Scrapy-splash not rendering dynamic content from a certain react-driven site Question: I am curious to see if any splash can get the dynamic job content from this page – https://nreca.csod.com/ux/ats/careersite/4/home?c=nreca#/requisition/182 in order for splash to receive the URL fragment you have to use a SplashRequest. In order for it to handle the JS cookies I …