When scraping content from a website, it seems like '&' in the URL is ignored Question: I have setup a web scraper, and I am now trying to add support for pagination. The URL changes as expected when going to the next page, and a ‘&page=X’ is added to the URL, X being the page …
When using BeautifulSoup to scrape Ebay, I get the wrong price for a listing Question: I am using BeautifulSoup to scrape the eBay site, however I got a different price than from the actual eBay page. I want to search for multiple different products later on and get the price for each of them, that’s …
How to extract links from a website in python? Question: I am trying to webscrape the below website. As a first step, I would like to get the links from which to extract the text. However, when I do the following, I get an empty list: import pandas as pd from bs4 import BeautifulSoup url …
Cannot locate text within using Python Question: Hi all, I am scraping questions on Amazon using the following code: url = "https://www.amazon.com/ask/questions/asin/B0000CFLYJ/1/ref=ask_ql_psf_ql_hza?isAnswered=true" r = requests.get("http://localhost:8050/render.html", params = {‘url’: url, ‘wait’: 3}) soup = BeautifulSoup(r.text, ‘html.parser’) questions = soup.find_all(‘div’, {‘class’:’a-fixed-left-grid-col a-col-right’}) print(questions) question_list = [] for item in questions: question = item.find(‘a’,{‘class’:’a-link-normal’}).text.strip() question_list.append(question) But I keep …
Attempting to download a .csv via a link on within a page w/ python Question: I’m attempting to download and use data contained in a .csv file. I’m pretty amatuer at scraping or most things coding related and would appreciate any help. Is it possible I’m being blocked from pulling the link from the website? …
Python requests result doesn't match the website because of JavaScript Question: I’m trying to scrape links of products from a webpage (url below). The page uses JavaScript. I tried different libraries, but the links don’t show up in the results (the links have the format */product/*, as you can see by hovering over product links …
Is there a way to scrape a page with XHR autoload? Question: there is this site with telegram chats of neighbours in Moscow. https://moscow.chatnovosela.ru/novostroyki i need to scrape it and get links to every card on this site. the trick is: cards are being appended by XHR when user is reaching the bottom of the …
Unable to scrape items using scrapy [solved] Question: I am trying to webscrape the name, price, and description of products listed on an online shop. The website link is https://eshop.nomin.mn/n-foods.html When I look through the HTML code of the page, I get the relevant div class containers but when I reference it in my code …
How to programmatically getting link to CSV behind javascript page? Question: I’m using python and I’m trying to get the link from which the CSV come from when I click on the DATA V CSV button at the bottom of this page. I tried beautifulsoup: import requests from bs4 import BeautifulSoup url = ‘https://www.ceps.cz/en/all-data#AktualniSystemovaOdchylkaCR’ response …
Scraping creators from a list of Youtube videos with certain amount of subscribers Question: This program is written in Python in the Selenium library, it generates a list of Youtube video links depending on what we typed in the search box. Is there any suggestion on how can I click on each one of these …
