OR condition in CSS Selector with Selenium/Python Question: I hope you’re fine. I’m scraping the logos of some websites. I’m using the next code to localize them. I don’t use a tag only the * because the class or attribute that contains the substring ‘logo’ there is not always in a <div> or <a> tags. …
Can't run Chrome in headless mode using Selenium Question: So here’s my code first: from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.chrome.options import Options import time from fake_useragent import UserAgent import random ua = UserAgent() options = Options() chrome_options = webdriver.ChromeOptions() chrome_options.add_argument(‘–blink-settings=imagesEnabled=false’) chrome_options.add_argument(‘–headless’) chrome_options.add_argument(f’user-agent={ua.random}’) driver = webdriver.Chrome(options=options, chrome_options=chrome_options) driver.maximize_window() url = "https://magiceden.io/marketplace/hasuki" driver.get(url) …
What is going on? (Attemp at scraping multiple pages) Question: url = "https://www.gumtree.com/search?search_category=all&q=ferrari" while url: response = requests.get(url) soup = BeautifulSoup(response.text, "html.parser") name = soup.find_all("div", class_="h3-responsive") price = soup.find_all("strong", "h3-responsive") next_page = soup.select_one("li.pagination-page>a") for price,name in zip(name,price): print(name.text,price.text) if next_page: next_url = next_page.get("href") url = urljoin(url,next_url) else: url = None Nothing is printing for some …
Run Scrapy from a script Question: I’m trying to run my script without the command "scrapy crawl…", I’m following this documentation https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script, but my code is not working. Would appreciate the help! import scrapy from scrapy.crawler import CrawlerProcess class misbeneficiosSpider(scrapy.Spider): name = ‘misbeneficios’ start_urls = [‘https://productos.misbeneficios.com.uy/tv-y-audio’, ‘https://productos.misbeneficios.com.uy/tv-y-audio?p=2’] def parse(self, response): for products in response.css(‘div.product-item-info’): yield …
Python – getting src from a table cell Question: I have a table as below for which I want to export the text OR the src to a csv file. <table class="GridView plm-table" id="pageLayout_projectTeamMembersGridView_gridView"> <tbody> <tr id="pageLayout_projectTeamMembersGridView_gridView_headerRow" class="GridViewHeaderRow"> <th class="GridViewHeader" scope="col">A</th> <th class="GridViewHeader" scope="col">B</th> <th class="GridViewHeader" scope="col">C</th> <th class="GridViewHeader" scope="col">D</th> <th class="GridViewHeader" scope="col">E</th> <th class="GridViewHeader" …
How can we use Mozilla to Screen Scrape raw data from real estate listings? Question: I’m looking at this URL. https://www.century21.com/real-estate/long-island-city-ny/LCNYLONGISLANDCITY/ I’m trying to get this text, in a structured format. FOR SALE $1,248,000 3 beds 2 baths 45-09 Skillman Avenue Sunnyside NY 11104 Listed By CENTURY 21 Sunny Gardens Realty, Inc. ########################################## FOR SALE …
Click on the button with Selenium, using the button text as a search element Question: I would like to click on the Google "News" button (after searching for something). I would like to search using the text News (in my case "Notizie") as an element. Google sometimes changes the names of the elements, so I …
Python Requests-html not return the page content Question: I’m new to Python and would like your advice for the issue I’ve encountered recently. I’m doing a small project where I tried to scrape a comic website to download a chapter (pictures). However, when printing out the page content for testing (because i tried to use …
Performing web scraping using selenium on influenster.com. I am getting just one scraped review even though it was in loop and the xpath was correct Question: from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from selenium.webdriver.common.by import By import configparser from datetime import datetime parser = configparser.RawConfigParser() parser.read(‘config.ini’) url= parser[‘PROPERTIES’][‘URL’] END_DATE = datetime.strptime(parser[‘DATE’][‘END’], …
Scraping Data frim AirBNB using Sellenium Question: Hi guys i am trying to scrape some data from airbnb in order to create a mini data analysis project for my portfolio. I tried several tutorials with BeautifulSoup but none of them is working today, even if I use the very same link that they are using …
