Data scraping of an aria-label with beautifulsoup

As @Ann Zen mentioned in commented the website is rendering elements and data dynamically and Beautifulsoup can’t handle it alone using Selenium will wait till the time app is loaded and then try to get the element

Example web-scraping-with-selenium

What about yfinance Python package? To get analyst price targets to you need to scroll the page to the end and wait until the data is loaded.

In this case, selenium library is used, which allows you to simulate user actions in the browser.

Install libraries:

pip install bs4 lxml selenium webdriver webdriver_manager

Import libraries:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import time, lxml

For selenium to work, you need to use ChromeDriver, which can be downloaded manually or using code. In our case, the second method is used. To control the start and stop of ChromeDriver, you need to use Service which will install browser binaries under the hood:

service = Service(ChromeDriverManager().install())

You should also add options to work correctly:

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--lang=en')
options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36')

• --headless – to run Chrome in headless mode.
• --lang=en – to set the browser language to English.
• user-agent – to act as a "real" user request from the browser by passing it to request headers. Check what’s your user-agent.

Now we can start webdriver and pass the URL to the get() method:

URL = 'https://finance.yahoo.com/quote/AAPL/analysis?p=AAPL'

driver = webdriver.Chrome(service=service, options=options)
driver.get(URL)

The page scrolling algorithm looks like this:

1. Find out the initial page height and write the result to the old_height variable.
2. Scroll the page using the script and wait 2 seconds for the data to load.
3. Find out the new page height and write the result to the new_height variable.
4. If the variables new_height and old_height are equal, then we complete the algorithm, otherwise we write the value of the variable new_height to the variable old_height and return to step 2.

Getting the page height and scroll is done by pasting the JavaScript code into the execute_script() method:

old_height = driver.execute_script("""
    function getHeight() {
        return document.querySelector('#Aside').scrollHeight;
    }
    return getHeight();
""")

while True:
    driver.execute_script('window.scrollTo(0, document.querySelector("#Aside").scrollHeight);')

    time.sleep(2)

    new_height = driver.execute_script("""
        function getHeight() {
            return document.querySelector('#Aside').scrollHeight;
        }
        return getHeight();
    """)

    if new_height == old_height:
        break

    old_height = new_height

We create the soup object and stop the driver:

soup = BeautifulSoup(driver.page_source, 'lxml')
driver.quit()

Selecting an element by class is not always a good idea because classes can change. It is more reliable to access attributes. In this case, I’m accessing the data-test attribute with the value price-targets and then to the div inside it. The value of the aria-label attribute is retrieved and printed from the resulting object:

price_targets = soup.select_one('[data-test="price-targets"] div').get('aria-label')
print(price_targets)

If you want to extract other data, you can see the Scrape Yahoo! Finance Home Page with Python blog post, which describes this in detail.

Code and full example in online IDE:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import time, lxml

URL = 'https://finance.yahoo.com/quote/AAPL/analysis?p=AAPL'

service = Service(ChromeDriverManager().install())

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--lang=en')
options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36')

driver = webdriver.Chrome(service=service, options=options)
driver.get(URL)

old_height = driver.execute_script("""
    function getHeight() {
        return document.querySelector('#Aside').scrollHeight;
    }
    return getHeight();
""")

while True:
    driver.execute_script('window.scrollTo(0, document.querySelector("#Aside").scrollHeight);')

    time.sleep(2)

    new_height = driver.execute_script("""
        function getHeight() {
            return document.querySelector('#Aside').scrollHeight;
        }
        return getHeight();
    """)

    if new_height == old_height:
        break

    old_height = new_height

soup = BeautifulSoup(driver.page_source, 'lxml')
driver.quit()

price_targets = soup.select_one('[data-test="price-targets"] div').get('aria-label')

print(price_targets)

Output:

Low  122 Current  129.62 Average  174.62 High  214