Python Scrape Weather Site Returns Nothing

Question

I’m trying to scrape a forecast table from wunderground. When inspecting the website, I see this:

Here is the code…When I run this, I get no results.

  import requests
  from bs4 import BeautifulSoup


   URL = "https://www.wunderground.com/calendar/us/ca/santa-barbara/KSBA"
   page = requests.get(URL)

   soup = BeautifulSoup(page.content, "html.parser")

   for ultag in soup.find_all('ul', {'class': 'calendar-days'}):

     for litag in ultag.find_all('li'):
        print (litag.text)

any help is appreciated.

Asked By: Allen

||

Source

Answer 1

As the site is using Javascript to generate the weather data, you will need to use something like Selenium to extract the required info. Using a combination of bs4 and Selenium – and with an appropriate sleep time, the weather data can be extracted.

Here is the code:

Code:

from bs4 import BeautifulSoup
from selenium import webdriver
from time import sleep

# Open browser and navigate to page
driver = webdriver.Chrome()
driver.get("https://www.wunderground.com/calendar/us/ca/santa-barbara/KSBA")

# Adjust sleep time to suit your needs
sleep(10)

html_source = driver.page_source
soup = BeautifulSoup(html_source, "html.parser")

# Find the weather data
for ultag in soup.find_all('ul', {'class': 'calendar-days'}):
    for litag in ultag.find_all('li'):
        print(litag.text)

driver.close()

Output:

27mostly sunnyActual:65° | 39°0 °in
28partly cloudyActual:62° | 48°0 °in
29mostly sunnyActual:61° | 40°0 °in
30cloudyActual:62° | 42°0 °in
1cloudyActual:59° | 53°0 °in
2cloudyActual:59° | 54°0 °in
3cloudyActual:64° | 57°0.56 °in
4cloudyActual:61° | 48°0 °in
5mostly sunnyActual:65° | 43°0.45 °in
6partly cloudyActual:61° | 41°0 °in
7partly cloudyActual:60° | 39°0 °in
8mostly sunnyActual:59° | 41°0 °in
9mostly sunnyActual:63° | 39°0 °in
10scattered showersActual:58° | 46°0 °in
11scattered showersActual:59° | 40°0.86 °in
12cloudyActual:56° | 42°0.25 °in
13mostly sunnyActual:58° | 33°0.14 °in
14mostly sunnyActual:59° | 37°0 °in
15mostly sunnyActual:58° | 42°0 °in
16mostly sunnyActual:65° | 41°0 °in
17mostly sunnyActual:62° | 37°0 °in
18mostly sunnyActual:61° | 35°0 °in
19mostly sunnyActual:61° | 38°0 °in
20mostly sunnyActual:62° | 40°0 °in
21mostly sunnyActual:64° | 41°0 °in
22mostly sunnyActual:65° | 41°0 °in
23mostly sunnyActual:64° | 42°0 °in
24mostly sunnyActual:74° | 41°0 °in
25mostly sunnyActual:78° | 43°0 °in
26mostly sunnyActual:72° | 44°0 °in
27scattered showersActual:60° | 53°0 °in
28mostly sunnyActual:62° | 49°1.01 °in
29RainForecast:60° | 50°0.1 °in
30Mostly CloudyForecast:62° | 51°0.05 °in

Answered By: ScottC

Python Scrape Weather Site Returns Nothing

Question:

Answers:

Code:

Output: