web-scraping

How can I store current directory as variable in python?

How can I store current directory as variable in python? Question: I’m trying to build a basic terminal that performs basic operations in python. I have made all the main functions, but the cd function isn’t working to change my current directory. I suspect that the problem is in the way I store my current …

Total answers: 3

Get Text from SVG using Python Selenium

Get Text from SVG using Python Selenium Question: My first time trying to extract data from an SVG element, following is the SVG element and the code I have tried to put up by reading stuff on the internet, I have absolutely no clue how wrong I am and why so. <svg class="rv-xy-plot__inner" width="282" height="348"> …

Total answers: 2

BeautifulSoup find a href in marquee

BeautifulSoup find a href in marquee Question: I’m using bs4 to scrape links from a scrolling marquee. I’m able to get the marquee data, which is returned as a bs4 resultSet element. However, I cannot seem to access the href’s within the data. I’m sure I’m missing something as I’m new to web scraping, and …

Total answers: 1

Find element with compound class in Selenium

Find element with compound class in Selenium Question: I can see some posts about this topic but unfortunately, none worked in my case. I am trying to locate elements with compounded classes in its name. This is the name of the elements class: class="group-header__wrapper is-grid-view-active section–prematch markets-optimized–3" I tried with this line of code, which …

Total answers: 1

BS4 not displaying text in Flask

BS4 not displaying text in Flask Question: I’m learning Python(Flask) and BeautifulSoup. For my first project I just wanted to wanted to get a video name from YT and display it on the homepage of my web app. An error returns: AttributeError: ‘NoneType’ object has no attribute ‘text’ import requests from flask import Blueprint, render_template …

Total answers: 1

reviews of a firm

reviews of a firm Question: My goal is to scrape the entire reviews of this firm. I tried manipulating @Driftr95 codes: def extract(pg): headers = {‘user-agent’ : ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36′} url = f’https://www.glassdoor.com/Reviews/3M-Reviews-E446_P{pg}.htm?filter.iso3Language=eng’ # f’https://www.glassdoor.com/Reviews/Google-Engineering-Reviews-EI_IE9079.0,6_DEPT1007_IP{pg}.htm?sort.sortType=RD&sort.ascending=false&filter.iso3Language=eng’ r = requests.get(url, headers, timeout=(3.05, 27)) soup = BeautifulSoup(r.content, ‘html.parser’)# this …

Total answers: 1

Tweets scraping using Python selinum

Tweets scraping using Python selinum Question: I am trying to scrape tweets under a hashtag using Python selinum and I use the following code to scroll down driver.execute_script(‘window.scrollTo(0,document.body.scrollHeight);’) The problem is that selinum only scrapes shown tweets (only 3 tweets) and then scroll down to the end of the page and load more tweets and …

Total answers: 2

Getting error when sending request to a website using Scrapy shell

Getting error when sending request to a website using Scrapy shell Question: I was learning Scrapy framework. I tried to use scrapy shell. There I was trying to fetch response from "https://quotes.toscrape.com/". The commands are below- python -m scrapy shell Inside the shell– >> from scrapy import Request >> req = Request("https://quotes.toscrape.com/") >> fetch(req) Then …

Total answers: 2

Selenium overlay button cannot be clicked

Selenium overlay button cannot be clicked Question: I am having some issues with one website’s button. Here is my driver function. def get_driver(): options = webdriver.ChromeOptions() # options.add_argument("–headless") options.add_argument("–incognito") driver = webdriver.Chrome(executable_path = ChromeDriverManager().install(), chrome_options = options) driver.get(‘https://online.depo-diy.lt’) return driver driver = get_driver() Problem is that I cannot select and click anything from the drop …

Total answers: 1

python selenium clicking a list object

python selenium clicking a list object Question: I am trying to click 1 Min button on this site below is my python code url = ‘https://www.investing.com/technical/technical-analysis’ driver.get(url) events = WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "section#leftColumn"))) print("Required elements found") events.find_element(By.XPATH,"//a[text()=’1 Min’]").click() Am getting the following error: events.find_element(By.XPATH,"//a[text()=’1 Min’]").click() AttributeError: ‘list’ object has no attribute ‘find_element’ What can I change …

Total answers: 2