Scraping stock price from Yahoo Finance using Python & BeautifulSoup

Question

I’m trying to scrape stock price from Yahoo Finance using Python and BeautifulSoup. However, I’m not able to fetch the tag having a specific data-reactid attribute (See the screenshot). Please help me.

Code:

def getCurrentPrice(self, stockSymbol):
    #stockSymbol is : MSFT for Microsoft
    
    url = "https://finance.yahoo.com/quote/{}".format(stockSymbol)
    source = requests.get(url).text
    soup = BeautifulSoup(source, 'lxml')

    currentPrice = soup.find('span',attrs={"data-reactid": "52"})

    print("{} : {}".format(stockSymbol, currentPrice))

Output:

MSFT : None # None because the span tag is not being found.

Asked By: David

||

Source

Answer 1

The attribute data-reactid is dynamic in nature so you can’t really find out the dom element using that.
Try this:

def getCurrentPrice(self, stockSymbol):
    #stockSymbol is : MSFT for Microsoft

    url = "https://finance.yahoo.com/quote/{}".format(stockSymbol)
    source = requests.get(url).text
    soup = BeautifulSoup(source, 'lxml')

    currentPrice = soup.find('span',attrs={"class": "Trsdu(0.3s)"})

    print("{} : {}".format(stockSymbol, currentPrice.get_text()))

Answered By: Shubham Sharma

Answer 2

You can use yfinance package to extract quote price in 3 lines:

import yfinance as yf

msft = yf.Ticker("MSFT")
print(msft.info["currentPrice"])

# 290.17

Besides changing selectors or dynamic selectors, the possible reason why you get None is because of not using user-agent in request headers to act as a "real" user visit.

Because default requests user-agent is python-requests and websites understand that it’s most likely a script that sends a request and might block it, thus you’ll receive a different HTML with some sort of an error that contains different selectors. Check what’s your user-agent.

Check code in online IDE.

import requests, lxml
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
}

html = requests.get("https://finance.yahoo.com/quote/MSFT", headers=headers)
soup = BeautifulSoup(html.text, "lxml")

price = soup.find("fin-streamer", class_="Fw(b) Fz(36px) Mb(-4px) D(ib)").text

print(price)

Example output


291.32

Answered By: Denis Skopa

Scraping stock price from Yahoo Finance using Python & BeautifulSoup

Question:

Answers: