python-requests

Web scraping reviews from Amazon only returns data for the first page

Web scraping reviews from Amazon only returns data for the first page Question: I am trying to scrape reviews from Amazon. The reviews can appear on multiple pages to scrape more than one page I construct a list of links which I later scrape separately: # Construct list of links to scrape multiple pages links …

Total answers: 3

How to download this specific img using python?

How to download this specific img using python? Question: You can open this link and you will see the image on your browser, you can right click on it and save it but you won’t be able to do it through a Python script, what am I missing here? I tried this simple approach and …

Total answers: 1

Python: BeautifulSoup select_one cannot find the tag

Python: BeautifulSoup select_one cannot find the tag Question: English is my second language, please excuse me for poor English. Follow code is an easy code that gets tag info with using requests and bs4. The problem is, this code is returning none. import requests from bs4 import BeautifulSoup url = ‘http://ch1.skbroadband.com/content/view?parent_no=24&content_no=57&p_no=154494’ web = requests.get(url,headers={‘User-Agent’:’Mozilla/5.0′}) source …

Total answers: 1

I can't find the correct tags to scrape the class name, code, and description (description is via link)

I can't find the correct tags to scrape the class name, code, and description (description is via link) Question: I’m brand new to scraping. I’m trying to scrape the class code, name, and description from this website: URL = https://catalog.registrar.ucla.edu/search?parentAcademicOrg=7e561ea0db6fa0107f1572f5f39619b1&ct=subject No matter what I set my divs = soup.find_all(), nothing seems to print (eventually I …

Total answers: 1

How to extract links from a website in python?

How to extract links from a website in python? Question: I am trying to webscrape the below website. As a first step, I would like to get the links from which to extract the text. However, when I do the following, I get an empty list: import pandas as pd from bs4 import BeautifulSoup url …

Total answers: 2

Python requests result doesn't match the website because of JavaScript

Python requests result doesn't match the website because of JavaScript Question: I’m trying to scrape links of products from a webpage (url below). The page uses JavaScript. I tried different libraries, but the links don’t show up in the results (the links have the format */product/*, as you can see by hovering over product links …

Total answers: 1

Is there a way to scrape a page with XHR autoload?

Is there a way to scrape a page with XHR autoload? Question: there is this site with telegram chats of neighbours in Moscow. https://moscow.chatnovosela.ru/novostroyki i need to scrape it and get links to every card on this site. the trick is: cards are being appended by XHR when user is reaching the bottom of the …

Total answers: 2

python lxml.etree generating empty tree when given requests.get().text

python lxml.etree generating empty tree when given requests.get().text Question: I am trying to build a web scraper for TEDTalks and am running into an issue when generating transcripts. Python 3.10.4, lxml 4.9.2. First I am generating the html response like this. text = requests.get(‘https://www.ted.com/talks/ted_countdown_how_do_we_get_the_world_off_fossil_fuels_quickly_and_fairly/transcript’, headers={}).text when checking the value of text, it shows me that …

Total answers: 1