Scrapy Playwright Page Method: Prevent timeout error if selector cannot be located Question: My question is related to Scrapy Playwright and how to prevent the Page of a Spider from crashing, if in the course of applying a PageMethod a specific selector cannot be located. Below is a Scrapy Spider that uses Playwright to interact …
How do I create a list from a webpage? Question: I am attempting to create a list of words from the website text. I would like to be able to randomise the word that is produced from this list using random. I hope this makes sense. import random as r from bs4 import BeautifulSoup import …
BeautifulSoup .find() only works on certain links Question: I have made a very simple web scraper which iterates through a list of links and scrapes the text and dates from them and outputs this into a text file. So far everything has been working fine but I have received an error which only occurs for …
How to extract price from web page using Beautiful Soup? Question: import requests from bs4 import BeautifulSoup URL="https://shop.beobasta.rs/proizvod/smrznuti-spanac/" header={"User Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 OPR/105.0.0.0","Accept-Language":"en-US,en;q=0.9"} response=requests.get(URL,headers=header) soup=BeautifulSoup(response.text,’html.parser’) price_element = soup.find("span",class_="woocommerce-Price-amount amount") print(price_element) I’m trying to extract price from this website but the only thing I’m getting is None. …
Trying to parse a beautifulsoap scrape for a specific row but I think it doesn't parse according to class or rowid — ex. I want text for rowid=2 Question: I am trying to get info (text) for rowid=2 (this should have narrowed the content significantly but I keep getting text for all requested td(s) or …
Scrapy : ValueError: XPath error: Invalid expression Question: I am trying to learn scrapy for a project. I receive this error ValueError: XPath error: Invalid expression but I don’t understand what is wrong in my script. It’s my script def parse(self, response): yield { ‘user_agent’: str(response.request.headers[‘User-Agent’]), ‘links’ : response.xpath(‘//a[@class="sc-996f251d-0 leAMGT"]/@href’).getall() } next = response.xpath(‘//a[@title="Page suivante"]/’) …
Have read_html read *cell content* and *tool tip* (text bubble) separately, instead of concatenate them Question: This site page has tooltips (text bubbles) appearing when hovering over values in columns "Score" and "XP LVL". It appears that read_html will concatenate cell content and tooltip. Splitting those in post-processing is not always obvious and I seek …
Trying to interact with HTML page elements, but none of them are found Question: I’m trying to scrape a webpage using Selenium, but when I try to pass the XPath of a button, I get an error saying that this element does not exist. I tried with another website, and it worked perfectly. I also …
Problem scraping Amazon using requests: I get blocked even when using cookie and headers. I can only scrape using a browser. Any solution? Question: The requests module isn’t working anymore for me when trying to scrape amazon, I’ve tried using cookies, headers, changing IP’s but nothing really works other than scraping through a browser. Does …
why did my web-scraping method stop working on one particular site? Question: several months ago I regularly used a python script to scrape and parse basketball odds from a particular website. after a couple months without using I tried to run the same script, only to find it now throws an error. I’m looking for …
