Datascraping using Beautiful soup. Finding the wrong body
Question:
I am trying to learning to web scrape. When I do ctrl + U to see the source code I don’t see any of the data I want to however if I inspect the page i can find the information in a different body.
I have written the following code:
url = "https://hereford.ttleagues.com/rankings/24018e59-6861-4082-bb62-5455224a3ad8"
result = requests.get(url)
doc = BeautifulSoup(result.text,"html.parser")
table1 = doc.find("body")
I find a body but not the one I am looking for.
The body I want to find
Is anybody able to tell me how to find the first body the has the information i want?
Answers:
The data that’s in the body you want comes from an API and is dynamically rendered.
You can query the API endpoint and rebuild the table.
For example:
import datetime
import pandas as pd
import requests
from tabulate import tabulate
headers = {
"Accept": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Host": "ttleagues-api.azurewebsites.net",
"Referer": "https://hereford.ttleagues.com/",
"Tenant": "hereford.ttleagues.com",
}
player_id = "24018e59-6861-4082-bb62-5455224a3ad8"
url = f"https://ttleagues-api.azurewebsites.net/api/rankings/{player_id}/stats"
results = requests.get(url, headers=headers).json()["results"]
stats = []
for result in results:
stats.append(
[
result["opponent"]["value"],
result["versus"]["value"],
datetime.datetime
.strptime(result["date"], "%Y-%m-%dT%H:%M:%SZ")
.strftime("%d %b %Y"),
result["for"],
result["against"],
" ".join(f"{g['for']}:{g['against']}" for g in result["games"]),
]
)
df = pd.DataFrame(
stats,
columns=["Opponent", "Versus", "Date", "For", "Against", "Game Scores"],
)
print(tabulate(df, headers="keys", tablefmt="psql", showindex=False))
Output:
+------------------------+---------------+-------------+-------+-----------+-----------------------------+
| Opponent | Versus | Date | For | Against | Game Scores |
|------------------------+---------------+-------------+-------+-----------+-----------------------------|
| Trevor Davies | Beekeepers | 08 Nov 2022 | 3 | 0 | 15:13 11:1 11:8 |
| Lincoln Franklin | Beekeepers | 08 Nov 2022 | 3 | 2 | 8:11 12:10 11:6 9:11 11:9 |
| Stephen Brown | Beekeepers | 08 Nov 2022 | 2 | 3 | 3:11 11:4 9:11 12:10 14:16 |
| Andrew Reece | Panthers | 25 Oct 2022 | 3 | 0 | 11:1 11:6 11:7 |
| Mark Playle | Panthers | 25 Oct 2022 | 3 | 0 | 11:6 11:8 11:5 |
| Harmesh Jutle | Panthers | 25 Oct 2022 | 3 | 1 | 11:4 8:11 11:2 11:9 |
| Paul Oatley | Warriors | 19 Oct 2022 | 3 | 1 | 11:5 11:8 7:11 11:8 |
| Toby Eckley | Warriors | 19 Oct 2022 | 0 | 3 | 9:11 8:11 7:11 |
| Ola Andersson | Warriors | 19 Oct 2022 | 1 | 3 | 5:11 3:11 12:10 7:11 |
| Andy Castle | Dodgers | 11 Oct 2022 | 3 | 0 | 11:8 11:7 11:6 |
| Alan Smith | Dodgers | 11 Oct 2022 | 3 | 0 | 11:8 11:4 11:8 |
| Sam Turner | Dodgers | 11 Oct 2022 | 3 | 2 | 12:14 11:9 14:12 13:15 11:8 |
| Patryk Kowacz | Pioneers | 06 Oct 2022 | 0 | 3 | 10:12 7:11 7:11 |
| Julian Clapp | Pioneers | 06 Oct 2022 | 3 | 0 | 11:5 11:5 11:4 |
| Piotr Sniezek | Pioneers | 06 Oct 2022 | 0 | 3 | 10:12 10:12 7:11 |
| Zak Weaver | Outcasts | 27 Sep 2022 | 3 | 2 | 7:11 11:9 9:11 11:6 11:7 |
| Dylan Day | Outcasts | 27 Sep 2022 | 2 | 3 | 11:5 10:12 10:12 11:4 10:12 |
| Simon Morgan | Outcasts | 27 Sep 2022 | 2 | 3 | 11:9 9:11 3:11 11:9 10:12 |
| Stephen Brown | Beekeepers | 22 Sep 2022 | 2 | 3 | 11:9 8:11 12:10 8:11 8:11 |
| Trevor Davies | Beekeepers | 22 Sep 2022 | 1 | 3 | 8:11 9:11 11:6 6:11 |
| Lincoln Franklin | Beekeepers | 22 Sep 2022 | 1 | 3 | 13:11 9:11 7:11 12:14 |
| Huan Davies | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 11:7 11:5 |
| Brian Caldicutt | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 11:7 11:5 |
| Andrew Reece | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 12:10 11:5 |
| Patryk Kowacz | Welly Wangers | 22 Feb 2022 | 2 | 3 | 8:11 13:15 11:6 11:9 5:11 |
| Nigel Foreshew | Welly Wangers | 22 Feb 2022 | 1 | 3 | 5:11 11:5 7:11 7:11 |
| Ken Read | Welly Wangers | 22 Feb 2022 | 0 | 3 | 9:11 8:11 8:11 |
| Radoslaw Mikolajczyk | Pioneers | 15 Feb 2022 | 1 | 3 | 11:7 7:11 8:11 5:11 |
| Yan Liu | Pioneers | 15 Feb 2022 | 3 | 0 | 11:3 11:3 11:9 |
| Harmesh Jutle | Pioneers | 15 Feb 2022 | 3 | 0 | 11:6 11:4 11:6 |
| Rod Davies | Magpies | 02 Feb 2022 | 1 | 3 | 11:7 9:11 7:11 7:11 |
| Toby Eckley | Magpies | 02 Feb 2022 | 2 | 3 | 12:10 4:11 11:3 6:11 7:11 |
| Paul Oatley | Magpies | 02 Feb 2022 | 3 | 0 | 11:5 11:7 11:8 |
| Waldemar Kogut | Panthers | 25 Jan 2022 | 1 | 3 | 11:9 8:11 9:11 8:11 |
| Pratikcha Chamling Rai | Panthers | 25 Jan 2022 | 3 | 0 | 11:1 13:11 11:6 |
| Harmesh Jutle | Panthers | 25 Jan 2022 | 3 | 0 | 11:7 11:7 11:8 |
| Ian Godding | Badgers | 19 Jan 2022 | 3 | 0 | 11:4 11:8 11:9 |
| Mark Conod | Badgers | 19 Jan 2022 | 3 | 1 | 11:2 13:11 8:11 12:10 |
| Jake Conod | Badgers | 19 Jan 2022 | 3 | 1 | 11:8 7:11 11:7 12:10 |
| Alan Smith | Dodgers | 11 Jan 2022 | 3 | 0 | 11:2 11:9 12:10 |
| Andy Castle | Dodgers | 11 Jan 2022 | 1 | 3 | 12:10 8:11 5:11 5:11 |
| Bryan Farr | Dodgers | 11 Jan 2022 | 3 | 0 | 11:9 11:8 11:6 |
| Dylan Day | Outcasts | 13 Dec 2021 | 1 | 3 | 11:9 3:11 7:11 13:15 |
| Zak Weaver | Outcasts | 13 Dec 2021 | 0 | 3 | 2:11 7:11 10:12 |
| Simon Morgan | Outcasts | 13 Dec 2021 | 2 | 3 | 8:11 5:11 11:6 14:12 1:11 |
| Bob Marshall | Commitments | 30 Nov 2021 | 3 | 0 | 11:8 11:7 11:4 |
| Pratikcha Chamling Rai | Commitments | 30 Nov 2021 | 3 | 0 | 11:2 11:5 11:8 |
| Brian Caldicutt | Commitments | 30 Nov 2021 | 3 | 0 | 11:4 11:7 11:4 |
| Toby Eckley | Magpies | 02 Nov 2021 | 2 | 3 | 11:6 11:13 14:12 15:17 8:11 |
| Chris Martin | Magpies | 02 Nov 2021 | 1 | 3 | 7:11 11:8 6:11 8:11 |
+------------------------+---------------+-------------+-------+-----------+-----------------------------+
I am trying to learning to web scrape. When I do ctrl + U to see the source code I don’t see any of the data I want to however if I inspect the page i can find the information in a different body.
I have written the following code:
url = "https://hereford.ttleagues.com/rankings/24018e59-6861-4082-bb62-5455224a3ad8"
result = requests.get(url)
doc = BeautifulSoup(result.text,"html.parser")
table1 = doc.find("body")
I find a body but not the one I am looking for.
The body I want to find
Is anybody able to tell me how to find the first body the has the information i want?
The data that’s in the body you want comes from an API and is dynamically rendered.
You can query the API endpoint and rebuild the table.
For example:
import datetime
import pandas as pd
import requests
from tabulate import tabulate
headers = {
"Accept": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Host": "ttleagues-api.azurewebsites.net",
"Referer": "https://hereford.ttleagues.com/",
"Tenant": "hereford.ttleagues.com",
}
player_id = "24018e59-6861-4082-bb62-5455224a3ad8"
url = f"https://ttleagues-api.azurewebsites.net/api/rankings/{player_id}/stats"
results = requests.get(url, headers=headers).json()["results"]
stats = []
for result in results:
stats.append(
[
result["opponent"]["value"],
result["versus"]["value"],
datetime.datetime
.strptime(result["date"], "%Y-%m-%dT%H:%M:%SZ")
.strftime("%d %b %Y"),
result["for"],
result["against"],
" ".join(f"{g['for']}:{g['against']}" for g in result["games"]),
]
)
df = pd.DataFrame(
stats,
columns=["Opponent", "Versus", "Date", "For", "Against", "Game Scores"],
)
print(tabulate(df, headers="keys", tablefmt="psql", showindex=False))
Output:
+------------------------+---------------+-------------+-------+-----------+-----------------------------+
| Opponent | Versus | Date | For | Against | Game Scores |
|------------------------+---------------+-------------+-------+-----------+-----------------------------|
| Trevor Davies | Beekeepers | 08 Nov 2022 | 3 | 0 | 15:13 11:1 11:8 |
| Lincoln Franklin | Beekeepers | 08 Nov 2022 | 3 | 2 | 8:11 12:10 11:6 9:11 11:9 |
| Stephen Brown | Beekeepers | 08 Nov 2022 | 2 | 3 | 3:11 11:4 9:11 12:10 14:16 |
| Andrew Reece | Panthers | 25 Oct 2022 | 3 | 0 | 11:1 11:6 11:7 |
| Mark Playle | Panthers | 25 Oct 2022 | 3 | 0 | 11:6 11:8 11:5 |
| Harmesh Jutle | Panthers | 25 Oct 2022 | 3 | 1 | 11:4 8:11 11:2 11:9 |
| Paul Oatley | Warriors | 19 Oct 2022 | 3 | 1 | 11:5 11:8 7:11 11:8 |
| Toby Eckley | Warriors | 19 Oct 2022 | 0 | 3 | 9:11 8:11 7:11 |
| Ola Andersson | Warriors | 19 Oct 2022 | 1 | 3 | 5:11 3:11 12:10 7:11 |
| Andy Castle | Dodgers | 11 Oct 2022 | 3 | 0 | 11:8 11:7 11:6 |
| Alan Smith | Dodgers | 11 Oct 2022 | 3 | 0 | 11:8 11:4 11:8 |
| Sam Turner | Dodgers | 11 Oct 2022 | 3 | 2 | 12:14 11:9 14:12 13:15 11:8 |
| Patryk Kowacz | Pioneers | 06 Oct 2022 | 0 | 3 | 10:12 7:11 7:11 |
| Julian Clapp | Pioneers | 06 Oct 2022 | 3 | 0 | 11:5 11:5 11:4 |
| Piotr Sniezek | Pioneers | 06 Oct 2022 | 0 | 3 | 10:12 10:12 7:11 |
| Zak Weaver | Outcasts | 27 Sep 2022 | 3 | 2 | 7:11 11:9 9:11 11:6 11:7 |
| Dylan Day | Outcasts | 27 Sep 2022 | 2 | 3 | 11:5 10:12 10:12 11:4 10:12 |
| Simon Morgan | Outcasts | 27 Sep 2022 | 2 | 3 | 11:9 9:11 3:11 11:9 10:12 |
| Stephen Brown | Beekeepers | 22 Sep 2022 | 2 | 3 | 11:9 8:11 12:10 8:11 8:11 |
| Trevor Davies | Beekeepers | 22 Sep 2022 | 1 | 3 | 8:11 9:11 11:6 6:11 |
| Lincoln Franklin | Beekeepers | 22 Sep 2022 | 1 | 3 | 13:11 9:11 7:11 12:14 |
| Huan Davies | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 11:7 11:5 |
| Brian Caldicutt | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 11:7 11:5 |
| Andrew Reece | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 12:10 11:5 |
| Patryk Kowacz | Welly Wangers | 22 Feb 2022 | 2 | 3 | 8:11 13:15 11:6 11:9 5:11 |
| Nigel Foreshew | Welly Wangers | 22 Feb 2022 | 1 | 3 | 5:11 11:5 7:11 7:11 |
| Ken Read | Welly Wangers | 22 Feb 2022 | 0 | 3 | 9:11 8:11 8:11 |
| Radoslaw Mikolajczyk | Pioneers | 15 Feb 2022 | 1 | 3 | 11:7 7:11 8:11 5:11 |
| Yan Liu | Pioneers | 15 Feb 2022 | 3 | 0 | 11:3 11:3 11:9 |
| Harmesh Jutle | Pioneers | 15 Feb 2022 | 3 | 0 | 11:6 11:4 11:6 |
| Rod Davies | Magpies | 02 Feb 2022 | 1 | 3 | 11:7 9:11 7:11 7:11 |
| Toby Eckley | Magpies | 02 Feb 2022 | 2 | 3 | 12:10 4:11 11:3 6:11 7:11 |
| Paul Oatley | Magpies | 02 Feb 2022 | 3 | 0 | 11:5 11:7 11:8 |
| Waldemar Kogut | Panthers | 25 Jan 2022 | 1 | 3 | 11:9 8:11 9:11 8:11 |
| Pratikcha Chamling Rai | Panthers | 25 Jan 2022 | 3 | 0 | 11:1 13:11 11:6 |
| Harmesh Jutle | Panthers | 25 Jan 2022 | 3 | 0 | 11:7 11:7 11:8 |
| Ian Godding | Badgers | 19 Jan 2022 | 3 | 0 | 11:4 11:8 11:9 |
| Mark Conod | Badgers | 19 Jan 2022 | 3 | 1 | 11:2 13:11 8:11 12:10 |
| Jake Conod | Badgers | 19 Jan 2022 | 3 | 1 | 11:8 7:11 11:7 12:10 |
| Alan Smith | Dodgers | 11 Jan 2022 | 3 | 0 | 11:2 11:9 12:10 |
| Andy Castle | Dodgers | 11 Jan 2022 | 1 | 3 | 12:10 8:11 5:11 5:11 |
| Bryan Farr | Dodgers | 11 Jan 2022 | 3 | 0 | 11:9 11:8 11:6 |
| Dylan Day | Outcasts | 13 Dec 2021 | 1 | 3 | 11:9 3:11 7:11 13:15 |
| Zak Weaver | Outcasts | 13 Dec 2021 | 0 | 3 | 2:11 7:11 10:12 |
| Simon Morgan | Outcasts | 13 Dec 2021 | 2 | 3 | 8:11 5:11 11:6 14:12 1:11 |
| Bob Marshall | Commitments | 30 Nov 2021 | 3 | 0 | 11:8 11:7 11:4 |
| Pratikcha Chamling Rai | Commitments | 30 Nov 2021 | 3 | 0 | 11:2 11:5 11:8 |
| Brian Caldicutt | Commitments | 30 Nov 2021 | 3 | 0 | 11:4 11:7 11:4 |
| Toby Eckley | Magpies | 02 Nov 2021 | 2 | 3 | 11:6 11:13 14:12 15:17 8:11 |
| Chris Martin | Magpies | 02 Nov 2021 | 1 | 3 | 7:11 11:8 6:11 8:11 |
+------------------------+---------------+-------------+-------+-----------+-----------------------------+