Using BeautifulSoup for finding scripts

Question

The code below was working as of a few days ago, but now it is only finding the first script for the url

url = 'https://understat.com/team/{}/2022'.format('Brentford')
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
scripts = soup.find_all('script')
scripts

Asked By: Luke

||

Source

Answer 1

Try to set a cookie when accessing the page:

import requests
from bs4 import BeautifulSoup

url = "https://understat.com/team/{}/2022".format("Brentford")

response = requests.get(url, cookies={"beget": "begetok"})  # <-- note the cookies= parameter
soup = BeautifulSoup(response.content, "html.parser")
scripts = soup.find_all("script")

print(scripts)

Prints:


...

                                        window.onload = function() { (adsbygoogle = window.adsbygoogle || []).push({}); }
                        </script>, <script defer="" src="js/date.format.min.js?v=2" type="text/javascript"></script>, <script defer="" src="js/calendar.js?v=2.1" type="text/javascript"></script>, <script defer="" src="js/team.js?v=2.5" type="text/javascript"></script>]

Answered By: Andrej Kesely

Using BeautifulSoup for finding scripts

Question:

Answers: