I'm trying to scrape a table web-site with different table option values

Question

I’m trying to scrap a table web-sites (an example is in this url: https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link ). But there are different option values in this table.

In particular I Would like to set the "Quarter View" option in: Q4 2021 13F Filings

The currently code implementing are:

test = {}
dict_scr = {}
for ii in range (0,12):
    options = webdriver.FirefoxOptions()
    options.binary_location = r'C://Local/Mozilla Firefox/firefox.exe'
    driver = selenium.webdriver.Firefox(executable_path='C:/geckodriver.exe' , options=options)
    driver.execute("get", {'url': link_scr['Links'][ii]})
    test[link_scr.index[ii]] = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table#current_holdings_table"))).get_attribute("outerHTML")
    dict_scr[link_scr.index[ii]]  = pd.read_html(test[link_scr.index[ii]])
    print(test[link_scr.index[ii]])

How I can update my code for including the option value required?

Asked By: Agustos Imola

||

Source

Answer 1

To select the <option> with text as Q1 2022 13F Filings you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.execute("get", {'url': 'https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.dfwid-close"))).click()
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "label#qtr-1-label"))))
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#quarter_one")))).select_by_value('85')

Using XPATH:

driver.execute("get", {'url': 'https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='dfwid-close']"))).click()
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//label[@id='qtr-1-label']"))))
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='quarter_one']")))).select_by_value('85')

Note : You have to add the following imports :

from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser snapshot:

Quarter to view

Answered By: undetected Selenium

I'm trying to scrape a table web-site with different table option values

Question:

Answers: