New to web scraping in Python and can't figure out my error

Question:

import requests
from bs4 import BeautifulSoup 
import pandas as pd

ticker = "TSLA"
url = "https://financialmodelingprep.com/financial-summary/" + ticker
request = requests.get(url)
print(request.text)

parser = BeautifulSoup(request.text, "html.parser")
news_html = parser.find_all('a', {'class': 'article-item'})
print(news_html[0])

sentiments = []
for i in range(0, len(news_html)):
    sentiments.append(
            {
                'ticker': ticker,
                'date': news_html[i].find('h5', {'class': 'article-date'}).text,
                'title': news_html[i].find('h4', {'class': 'article-title'}).text,
                'text': news_html[i].find('p', {'class': 'article-text'}).text
            }
        )

df = pd.DataFrame(sentiments)
df = df.set_index('date')

---------------------------------------------------------------------------

    IndexError                                Traceback (most recent call last)
    Input In [6], in <cell line: 12>()
         10 parser = BeautifulSoup(request.text, "html.parser")
         11 news_html = parser.find_all('a', {'class': 'article-item'})
    ---> 12 print(news_html[10])
         14 sentiments = []
         15 for i in range(0, len(news_html)):
    
    IndexError: list index out of range

I am trying to scrape data for sentiment analysis. There is a second part of the code that is supposed to calculate a sentiment score but I cannot get past the error.

Asked By: Eddie Chinea

||

Answers:

import requests
from bs4 import BeautifulSoup 
import pandas as pd

ticker = "TSLA"
url = "https://financialmodelingprep.com/financial-summary/" + ticker
request = requests.get(url)

parser = BeautifulSoup(request.text, "html.parser")
news_html = parser.find('div', {'class':'articles'})

sentiments = []
for div in news_html.find_all('a', {'class':'article'}):
    try:
        sentiments.append(
                {
                    'ticker': ticker,
                    'date': div.find('h5', {'class': 'article__date'}).text,
                    'title': div.find('div', {'class': 'article__title'}).text,
                    'text': div.find('p', {'class': 'article__text'}).text
                })
        
    except Exception:
        pass
Answered By: A259
Categories: questions Tags:
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.