Can't scrape table BeautifulSoup

Question

I’m trying to scrape the following table from this URL: https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show

This is my code:

import requests
from bs4 import BeautifulSoup

url = "https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show"
r = requests.get(url)

soup = BeautifulSoup(r.content, "lxml")

table = soup.find("table")
for row in table.findAll("tr"):
    print([i.text for i in row.findAll("td")])

However, my variable table returns None, even though there is clearly a table tag in the HTML code of the website. How do I get it?

Asked By: earningjoker430

||

Source

Answer 1

The webpage is loaded dynamically and relies on JavaScript, therefore requests won’t support it. You could use another parser library such as selenium.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()

url = "https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show"

driver.get(url)

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.TAG_NAME, 'table')))

table = driver.find_element(By.TAG_NAME, 'table')

table_html = table.get_attribute('innerHTML')

# print('table html:', table_html)

for tr_web_element in table.find_elements(By.TAG_NAME, 'tr'):
    for td_web_element in tr_web_element.find_elements(By.TAG_NAME, 'td'):
        print(td_web_element.text)

driver.close()

Or see this answer to incorporate Selenium with BeautifulSoup.

Answered By: idkwhatimdoing

Can't scrape table BeautifulSoup

Question:

Answers: