How to click 'View' button using Selenium on Wunderground Historical Weather page

Question:

I’m trying to access the "View" button on the "https://www.wunderground.com/history" website using Selenium in Python but I’m unable to do so. I’ve tried multiple methods including:

  • ActionChains("actions.move_to_element(button_view).click().perform()")
  • browser.find_element(By.ID, ‘dateSubmit’).click()
  • wait.until(EC.element_to_be_clickable((By.ID, ‘dateSubmit’)))
  • actions.move_to_element(button_view).double_click(button_view)

but none of them seems to work.
Interestingly, when I tried to access a similar location page (https://www.wunderground.com/history/daily/KLGA/), all I needed to do was type a search in the ‘historySearch’ box and the button would click, but on the "/history" page, it wouldn’t work no matter what I did.

I suspect the reason for this is due to the different actions performed before clicking the button, such as clicking the search text box area, entering a location (e.g. "New York, NY"), and selecting an option from the auto-completion dropdown menu.

In my case, I couldn’t select the desired option from the dropdown menu, so I tried to unfocus from the text box, which in the site’s case automatically chooses the first auto-completion option for you.

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from datetime import date, timedelta
from selenium.webdriver.common.by import By
from selenium.webdriver import ActionChains
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
from collections import Counter
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup
from datetime import datetime

import time
import logging
import pandas as pd

def get_weather_data4( year, month, day, browser):
    try:
        # Open browser and navigate to the website
        #browser = webdriver.Chrome()
        wait = WebDriverWait(browser, 20)
        browser.get('https://www.wunderground.com/history')
        wait.until(EC.presence_of_element_located((By.ID, 'yearSelection')))
        wait.until(EC.presence_of_element_located((By.ID, 'historySearch')))
        
        # Select year, month, and day from the drop-down menus
        actions = ActionChains(browser)
        year_select = Select(browser.find_element(By.ID,'yearSelection'))
        year_select.select_by_value(str((2023-year))+':'+' '+str(year))
        month_select = Select(browser.find_element(By.ID,'monthSelection'))
        month_select.select_by_value(str(month))
        day_select = Select(browser.find_element(By.ID,'daySelection'))
        day_select.select_by_value(str((day-1))+':'+' '+str(day))
     
        search_field = wait.until(EC.visibility_of_element_located((By.ID, 'historySearch')))
        search_field.click()
        search_field.send_keys("New York, NY", Keys.ENTER)
        #search_field.send_keys(Keys.TAB * 2 )#+ Keys.RETURN * 2)
     

        try:
            wait.until(EC.element_to_be_clickable((By.ID, 'dateSubmit')))
            button_view = browser.find_element(By.ID, 'dateSubmit')
            #button_view.send_keys("",Keys.ENTER)
            actions.move_to_element(button_view).double_click(button_view)
        except TimeoutError as te:
            print(f"The following element caused a timeout error: {te}")
       
    # Click the "View" button to load the data for the selected date FAILED ATTEMPTS
        #button_view = wait.until(EC.element_to_be_clickable((By.ID, 'dateSubmit')))
        #actions.move_to_element(button_view).click().perform()
        #button_view.click()
        #browser.find_element(By.ID, 'dateSubmit').click()
        #WebDriverWait(browser, 2)
        #browser.find_element(By.ID, 'dateSubmit').click()
        
        # Wait for the data to load
        wait.until_not(EC.presence_of_element_located((By.XPATH, '//div[@class="loading-indicator"]')))
        wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'row')))
        wait.until(EC.presence_of_element_located((By.XPATH, '//table[@class="mat-table cdk-table mat-sort ng-star-inserted"]')))
    
    except TimeoutError:
        raise Exception("Timeout Error: Could not load data from URL")
    
    # Extract the data from the table using BeautifulSoup
    soup = BeautifulSoup(browser.page_source, 'html.parser')
    table = soup.select_one('table[class*="mat-table cdk-table mat-sort ng-star-inserted"]')
    rows = table.find_all('tr',{'class':'mat-row cdk-row ng-star-inserted'})
  # Code continues... 

Relevant Details:

  • Selenium version: 4.7.2
  • Browser: Firefox
  • Operating System: Windows 10

Request: I am seeking assistance in resolving this issue and clicking the "View" button on the website. Any help or suggestions would be greatly appreciated.

Asked By: Shai Shillo

||

Answers:

To fill up the Location box with Boston, click on the first autocomplete item and click on the element View you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategies:

driver.get('https://www.wunderground.com/history')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#historySearch[name='historySearch']"))).send_keys("Boston")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//search-autocomplete//ul[@class='ui-autocomplete ui-front ui-menu ui-widget ui-widget-content ui-corner-all']//li/a/span[contains(., 'MA')]/span[text()='Boston']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#dateSubmit[value='View']"))).click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser Snapshot:

EastBoston

Answered By: undetected Selenium