Python BeautifulSoup4 news scraper giving odd error

Question

I am trying to make a news scraper with BS4 and I am able to get the html code from the website (cnn) and this is my code:

from bs4 import BeautifulSoup
import requests

url = "https://www.cnn.com/"
topic = input("What kind of news are you looking for")
result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

prices = doc.find_all(text = f"{topic}")
parent = prices[0].parent
print(parent)

but its giving me this error

xxx@xxx-xxx xxx % python3 news_scraper.py
What kind of news are you looking for?Coronavirus
Traceback (most recent call last):
  File "/xxx/xxx/xxx/xxx/news_scraper.py", line 10, in <module>
    parent = prices[0].parent
IndexError: list index out of range

I have no idea what is causing this, Thanks!

Asked By: yavda

||

Source

Answer 1

If the string topic is not found on the page, then prices will be an empty array. To fix this, first check that the length of prices is not zero. Like this:

from bs4 import BeautifulSoup
import requests

url = "https://www.cnn.com/"
topic = input("What kind of news are you looking for")
result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")
prices = doc.find_all(text = f"{topic}")

if len(prices) != 0:
     parent = prices[0].parent
     print(parent)
else:
     print("No news of that topic was found.");

Answered By: Michael M.

Answer 2

I think the issue is that most of CNN’s page is dynamic and beautifulsoup can’t read dynamically generated content. The sections at the bottom of the page are native to that url and it works just fine on those. You need something like Selenium for dynamic pages.

Answered By: RedKnite

Answer 3

doc = BeautifulSoup(result.text, "html.parser")
print(doc)

Add print and look there not DOM, only JS and CSS

Answered By: Сеня Колышкин

Python BeautifulSoup4 news scraper giving odd error

Question:

Answers: