New to Python and Web-scraping. Scraping an HTML table- however it's not displaying all columns

Question

I’m using BeautifulSoup and trying to scrap a HTML table. I’m only interested in the first table. However, the output is missing one column values – the "Entries" column. Not sure what I’m doing wrong here.

Here’s my code:

import requests
from bs4 import BeautifulSoup


page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

table = soup.find_all('table')[0]

for child in soup.find_all('table')[0].children:
    for td in child:
        print(td.text)

Asked By: Ventorro

||

Source

Answer 1

The easiest way how to read the first table is using pandas.read_html:

import pandas as pd

url = "http://www.godaycare.com/child-care-cost/saskatchewan"

df = pd.read_html(url)[0]
print(df.to_markdown())

Prints:

	Type	Age Cat.	Spot	AVG. Cost ($)	Entries
0	Licensed	Infant	Full-Time	751.02	717
1	Licensed	Infant	Part-Time	41.31	187
2	Unlicensed	Infant	Full-Time	699.56	287
3	Unlicensed	Infant	Part-Time	31.05	50
4	Licensed	Toddler	Full-Time	661.04	604
5	Licensed	Toddler	Part-Time	32.69	148
6	Unlicensed	Toddler	Full-Time	633.01	342
7	Unlicensed	Toddler	Part-Time	35.99	69
8	Licensed	Preschool	Full-Time	595.45	327
9	Licensed	Preschool	Part-Time	30.85	66
10	Unlicensed	Preschool	Full-Time	602.82	195
11	Unlicensed	Preschool	Part-Time	30.33	30
12	Licensed	Kindergarten	Full-Time	562.87	87
13	Licensed	Kindergarten	Part-Time	28.29	38
14	Unlicensed	Kindergarten	Full-Time	549.12	57
15	Unlicensed	Kindergarten	Part-Time	23.01	13
16	Licensed	Schoolage	Full-Time	605.34	94
17	Licensed	Schoolage	Part-Time	25.45	33
18	Unlicensed	Schoolage	Full-Time	434.9	98
19	Unlicensed	Schoolage	Part-Time	19	25

EDIT: Version with beautifulsoup:

import requests
from bs4 import BeautifulSoup

URL = "http://www.godaycare.com/child-care-cost/saskatchewan"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

for row in soup.find("table").find_all("tr"):
    tds = [td.text for td in row.find_all(["td", "th"])]
    print(("{:<20}" * len(tds)).format(*tds))

Prints:

Type                Age Cat.            Spot                AVG. Cost ($)       Entries             
Licensed            Infant              Full-Time           751.02              717                 
Licensed            Infant              Part-Time           41.31               187                 
Unlicensed          Infant              Full-Time           699.56              287                 
Unlicensed          Infant              Part-Time           31.05               50                  
Licensed            Toddler             Full-Time           661.04              604                 
Licensed            Toddler             Part-Time           32.69               148                 
Unlicensed          Toddler             Full-Time           633.01              342                 
Unlicensed          Toddler             Part-Time           35.99               69                  
Licensed            Preschool           Full-Time           595.45              327                 
Licensed            Preschool           Part-Time           30.85               66                  
Unlicensed          Preschool           Full-Time           602.82              195                 
Unlicensed          Preschool           Part-Time           30.33               30                  
Licensed            Kindergarten        Full-Time           562.87              87                  
Licensed            Kindergarten        Part-Time           28.29               38                  
Unlicensed          Kindergarten        Full-Time           549.12              57                  
Unlicensed          Kindergarten        Part-Time           23.01               13                  
Licensed            Schoolage           Full-Time           605.34              94                  
Licensed            Schoolage           Part-Time           25.45               33                  
Unlicensed          Schoolage           Full-Time           434.90              98                  
Unlicensed          Schoolage           Part-Time           19.00               25

Answered By: Andrej Kesely

New to Python and Web-scraping. Scraping an HTML table- however it's not displaying all columns

Question:

Answers: