Why does soup.findAll return []?

Question

I am having trouble reading data from a url with BeautifulSoup

This is my code:

url1 = "https://www.barstoolsportsbook.com/events/1018282280"
from bs4 import BeautifulSoup
import requests

headers = {
    'User-agent':
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

html = requests.get(url1, headers=headers)
soup = BeautifulSoup(html.text, 'lxml')
data = soup.findAll('div',attrs={"class":"section"})
print(data)
#for x in data:
#    print(x.find('p').text)

When I print(data) I am returned []. What could be the reason for this? I would like to avoid using selenium for this task if possible.

This is the HTML for what I’m trying to grab

<div data-v-50e01018="" data-v-2a52296d="" class="section"><p data-v-5f665d29="" data-v-50e01018="" class="header strongbody2">HOT TIPS</p><p data-v-5f665d29="" data-v-50e01018="" class="tip body2"> The Mets have led after 3 innings in seven of their last nine night games against NL East Division opponents that held a losing record. </p><p data-v-5f665d29="" data-v-50e01018="" class="tip body2"> The 'Inning 1 UNDER 0.5 runs' market has hit in each of the Marlins' last nine games against NL East opponents. </p></div>

Asked By: smitty_werben_jagerm

||

Source

Answer 1

You can likely get what you want with this request:

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0',
    'Accept': 'application/json, text/plain, */*',
    'Accept-Language': 'en-US,en;q=0.5',
    'Authorization': 'Basic MTI2OWJlMjItNDI2My01MTI1LWJlNzMtMDZmMjlmMmZjNWM3Omk5Zm9jajRJQkZwMUJjVUc0NGt2S2ZpWEpremVKZVpZ',
    'Origin': 'https://www.barstoolsportsbook.com',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Referer': 'https://www.barstoolsportsbook.com/',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'cross-site',
}

response = requests.get('https://api.isportgenius.com.au/preview/1018282280', headers=headers)
response.json()

You should browse the network tab to see where the rest of the data is coming from, or use a webdriver.

Answered By: 1extralime

Why does soup.findAll return []?

Question:

Answers: