How to get weather data from site?

bs4’s BeautifulSoup is often used to parse html and extract data. In your case, it can be utilized in several ways; first parse and get the html tree [soup] with

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://weather.gc.ca/city/pages/ab-52_metric_e.html")
soup = BeautifulSoup(resp.content, "html.parser")

[I like using the select function, and you can consult this reference if you’re confused by any of the selectors.]

---

Now, if you want the current weather, then

selector = '#mainContent details.visible-xs dt+dd'
for s in soup.select(selector):
    print(s.find_previous_sibling().text.strip(), s.text.strip(), end=' ')

will output

Wind: NE 8 km/h Temperature: 18.2°C Pressure: 101.9 kPa Dew point: -8.1°C Visibility: 24 km Humidity: 16% Date: 8:00 PM MDT Wednesday 19 October 2022 Observed at: Calgary Int'l Airport 

but you also commented about wanting the Condition and Tendency, and you could get the Condition simply with

print(f"Condition: {soup.select_one('details.visible-xs div>img+p')}")

but Tendency is actually in a different section which is not visible. You could change selector to '#mainContent section:first-of-type dt+dd' or even to just 'dt+dd' and run for s in soup.select(selector).... again, but you will several repeated values and unwanted data, so we probably need a more structured approach.

---

---

So I defined a function that will take a section of the tree and return one value per condition –

def getWeatherData(sectionSoup, pKeys='all', cwcSelector='dt+dd', pEnd=' '):
    wc = []
    for s in sectionSoup.select(cwcSelector):
        wcName = s.find_previous_sibling().text.strip()
        wcVal = s.text.strip()

        vis = 'hidden' if s.find_parent(class_='hidden-xs') else 'visible'
        wc.append((wcName, wcVal, vis)) 

    wcs = sorted(wc, key=lambda c: c[2], reverse=True) # visible first
    if pKeys == 'all' or type(pKeys) != list: 
      pKeys = list(set([c[0] for c in wc]))
      allKeys = True
    else: allKeys = False

    forOp = []
    for k in pKeys:
      kvp = [c for c in wcs if c[0].replace(':','')==k.replace(':','')]
      if kvp == []:
          # continue # if you want to skip
          if pEnd is None or type(pEnd) == str: 
              print(f'! UNAVAILABLE : "{k}" !', end=pEnd)
      else: 
          if allKeys: k = k.replace(':', '') # remove if you want to preserve originial text
          forOp.append((k, kvp[0][1]))
          if pEnd is None or type(pEnd) == str: 
              print(kvp[0][0], kvp[0][1], end=pEnd) 
      
    print()
    return dict(forOp)
    
    # if you want all the values, including repeats:
    # return {'filtered': forOp, 'unfiltered': wc} 

Now you can define which parts of the data you want, and what order you want it in:

toPrint = [
    'Condition', 'Pressure', 'Tendency', 'Temperature', 
    'Dew point', 'Humidity', 'Wind', 'Visibility'
] # the parts you mentioned wanting

cwc = getWeatherData(soup.select_one('#mainContent section'), toPrint)
print(f'n##############nAs Dictionary:n{cwc}')

and that will output

Condition: Partly Cloudy Pressure: 101.3 kPa Tendency: Rising Temperature: 8.9°C Dew point: -0.3°C Humidity: 52% Wind: N 43  gust 54 km/h Visibility: 24 km 

##############
As Dictionary:
{'Condition': 'Partly Cloudy', 'Pressure': '101.3 kPa', 'Tendency': 'Rising', 'Temperature': '8.9°C', 'Dew point': '-0.3°C', 'Humidity': '52%', 'Wind': 'N 43  gust 54 km/h', 'Visibility': '24 km'}

[Send pEnd='n' if you want each value printed on a separate line, or pEnd=False if you don’t want to print at all; and send pKeys='all' (instead of toPrint) if you want to see all available data.]

---

---

If you wanted, you could also see all collapsible sections (hidden and visible) with

wcSects = [s.find_parent('details') for s in soup.select('details dd+dt')]

for s in list(set(wcSects)):
  print('n#######', end=' ')  
  h2 = s.select_one('summary h2')
  print((h2 if h2 else s.summary).text.strip(), '#######')
  getWeatherData(s, 'all', pEnd=' | ')