scrape a specific div value with beautifulsoup in nested div

Question

I currently try scrape a value at this specific website for a school project https://data.census.gov/cedsci/table?q=53706%20income&tid=ACSST5Y2020.S1901
it’s the first one below if you search Median income (dollars), which should be the median income of the area, the comp-id keep changing for some reason

This median income estimate is what I’m looking for

I tried serveral method on the sites to go over the nested divs but I’m not able to get any results after runned, below is a code that I tried to use, but it just kept returning nothing to me.
Any help will be appreciate, thanks!

import csv
import requests
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import pandas as pd
from bs4 import BeautifulSoup 



DRIVER_PATH = 'chromedriver_107.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)


            url = 'https://data.census.gov/cedsci/table?q=' + '53706' + '%20income&tid=ACSST5Y2020.S1901'
            driver.get(url)

            page = requests.get(url)


            content = driver.page_source
            soup = BeautifulSoup(content, 'lxml')

            a = soup.findAll("div", {"comp-id":"1539"})
            print(a)

Asked By: Zhe Wang

||

Source

Answer 1

Try with this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#set up Chrome driver
options=webdriver.ChromeOptions()


#Define web driver as a Chrome driver and navigate
driver = webdriver.Chrome()
driver.maximize_window()

url = 'https://data.census.gov/cedsci/table?q=53703%20income&tid=ACSST5Y2020.S1901'
driver.get(url)

# We print the label of row 11 (Which is the median)
label = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "(//div[@row-id='11'])[1]")))
print(label.text)
# We print the values of row 11 (Which is the median)
values = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "(//div[@row-id='11'])[2]")))
print(values.text)

Output:

Median income (dollars)
42,153
±3,200
114,643
±28,572
139,694

Answered By: Jaky Ruby

scrape a specific div value with beautifulsoup in nested div

Question:

Answers: