Selenium: Selecting from Multiple Drop-Downs at Once

Question:

I am building a web scraper that has to try a combination of multiple drop-down menu options and gather data from each combination.
So basically there’re 5 drop-downs. I have to gather data from all of the possible combinations of the drop-down options. For each combination, I have to press a button to pull up the page with all the data on it. I am storing all the data in a dictionary.

This is the website: http://siops.datasus.gov.br/filtro_rel_ges_covid_municipal.php?S=1&UF=12;&Municipio=120001;&Ano=2020&Periodo=20

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select

# General Stuff about the website
path = '/Users/admin/desktop/projects/scraper/chromedriver'
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")
driver = webdriver.Chrome(options=options, executable_path=path)
website = 'http://siops.datasus.gov.br/filtro_rel_ges_covid_municipal.php'
driver.get(website)

# Initial Test: printing the title
print(driver.title)
print()

# Dictionary to Store stuff in
totals = {}

# Drop Down Menus
year_select = Select(driver.find_element(By.XPATH, '//*[@id="cmbAno"]'))
uf_select = Select(driver.find_element(By.XPATH, '//*[@id="cmbUF"]'))

### THIS IS WHERE THE ERROR IS OCCURING ###
# Choose from the drop down menus 
uf_select.select_by_value('29')
year_select.select_by_value('2020')

# Submit button on the page
submit_button = driver.find_element(By.XPATH, '//*[@id="container"]/div[2]/form/div[2]/div/input[2]')
submit_button.click()

# Pulling data from the webpage
nameof = driver.find_element(By.XPATH, '//*[@id="arearelatorio"]/div[1]/div/table[1]/tbody/tr[2]').text
total_balance = driver.find_element(By.XPATH, '//*[@id="arearelatorio"]/div[1]/div/table[3]/tbody/tr[9]/td[2]').text
paid_expenses = driver.find_element(By.XPATH, '//*[@id="arearelatorio"]/div[1]/div/table[4]/tbody/tr[11]/td[4]').text

# Update Dictionary with the new info
totals.update({nameof: [total_balance, paid_expenses]})
totals.update({'this is a test': ['testing stuff']})

# Print the final Dictionary and quit
print(totals)
driver.quit()

For some reason, this code does not work when trying 1 possible combination (selecting value 29 from the UF drop-down, as well as value 2020 from the year_select drop-down). If I comment out of the two drop-down selections, then it works perfectly fine.

How do I try multiple combinations of drop-down options during a single iteration?

Asked By: Seaborg

||

Answers:

try this instead.

# Drop Down Menus

### THIS IS WHERE THE ERROR IS OCCURING ###
# Choose from the drop down menus
uf_select = Select(driver.find_element(By.XPATH, '//*[@id="cmbUF"]')) 
uf_select.select_by_value('29')

year_select = Select(driver.find_element(By.XPATH, '//*[@id="cmbAno"]'))
year_select.select_by_value('2020')

This works for me. With your example i get a stale… error, means that the element disappears. HavenĀ“t checked, but maybe the checkbox is somehow updated and looses reference when selecting the other one.

Answered By: Daemon