web-scraping

How do I create a list from a webpage?

How do I create a list from a webpage? Question: I am attempting to create a list of words from the website text. I would like to be able to randomise the word that is produced from this list using random. I hope this makes sense. import random as r from bs4 import BeautifulSoup import …

Total answers: 2

BeautifulSoup .find() only works on certain links

BeautifulSoup .find() only works on certain links Question: I have made a very simple web scraper which iterates through a list of links and scrapes the text and dates from them and outputs this into a text file. So far everything has been working fine but I have received an error which only occurs for …

Total answers: 1

How to extract price from web page using Beautiful Soup?

How to extract price from web page using Beautiful Soup? Question: import requests from bs4 import BeautifulSoup URL="https://shop.beobasta.rs/proizvod/smrznuti-spanac/" header={"User Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 OPR/105.0.0.0","Accept-Language":"en-US,en;q=0.9"} response=requests.get(URL,headers=header) soup=BeautifulSoup(response.text,’html.parser’) price_element = soup.find("span",class_="woocommerce-Price-amount amount") print(price_element) I’m trying to extract price from this website but the only thing I’m getting is None. …

Total answers: 2

Scrapy : ValueError: XPath error: Invalid expression

Scrapy : ValueError: XPath error: Invalid expression Question: I am trying to learn scrapy for a project. I receive this error ValueError: XPath error: Invalid expression but I don’t understand what is wrong in my script. It’s my script def parse(self, response): yield { ‘user_agent’: str(response.request.headers[‘User-Agent’]), ‘links’ : response.xpath(‘//a[@class="sc-996f251d-0 leAMGT"]/@href’).getall() } next = response.xpath(‘//a[@title="Page suivante"]/’) …

Total answers: 1

Trying to interact with HTML page elements, but none of them are found

Trying to interact with HTML page elements, but none of them are found Question: I’m trying to scrape a webpage using Selenium, but when I try to pass the XPath of a button, I get an error saying that this element does not exist. I tried with another website, and it worked perfectly. I also …

Total answers: 1

why did my web-scraping method stop working on one particular site?

why did my web-scraping method stop working on one particular site? Question: several months ago I regularly used a python script to scrape and parse basketball odds from a particular website. after a couple months without using I tried to run the same script, only to find it now throws an error. I’m looking for …

Total answers: 2

Panda not printing all of the table

Panda not printing all of the table Question: This is my first post so I hope I don’t forget anything. So I was trying to scrape all of the UFC events to see certain stats of fighters and I tried using Pandas. This is where my problem started, so when I imported the website using …

Total answers: 1

Selenium Python: How to capture li element with specific text

Selenium Python: How to capture li element with specific text Question: I am trying to extract urlToBeCaptured and Text to be captured from the HTML. The structure looks as follows: <li> " text with trailing spaces " <a href="urlToBeCaptured"> <span class ="class1> Text to be captured </span> <span class ="class2> Another text </span> </a> … …

Total answers: 1