Python webscraper with Beautiful soup only returns one row of data

Question

I’m new to coding and for a class project, we’re supposed to build a webscraper using python and BeautifulSoup. Following some tutorials I’ve found online, I put together this code that seemingly works in general, but for the print(nick) command only returns the last row of the referenced table. Weirdly enough, the print(table) command is able to return all rows/all the table data.

One more issue the script has is that it can’t return the "team" data which is stored differently in the HTML table:

I’m a beginner, so I’m having trouble understanding what’s causing this issue.

import requests
from bs4 import BeautifulSoup

url = 'https://www.hltv.org/stats/players?startDate=2022-02-24&endDate=2023-02-24&matchType=Lan'

#empty array
player_list = []

#requesting and parsing the HTML file
response = requests.get(url)
#print(response.status_code)
soup = BeautifulSoup(response.text, 'html.parser')

#selecting the table
table = soup.find('table', class_ = 'stats-table')
###print(table)

#storing all rows into one variable
for player_data in table.find_all('tbody'):
    rows = player_data.find_all('tr')

#looping through the HTML table to scrape the data
    for row in rows:
        nick = row.find_all('td')[0].text
        team = row.find_all('td')[1].text
        maps = row.find_all('td')[2].text
        rounds = row.find_all('td')[3].text
        kddiff = row.find_all('td')[4].text
        killdeath = row.find_all('td')[5].text
        rating = row.find_all('td')[6].text

print(nick)
print(team)

If anyone’s able to help, please let me know!

Asked By: quarantinho

||

Source

Answer 1

First, you output the last value stored in the nick variable, print(nick) must be added to the loop to output all the names. As for print(team), there are several more tags in the ‘td’ tag referring to the names of the ‘img’ commands, you need to find them all, then iterate through the loop and return the entire name using get(), here is an example of an implementation with the output of these values:

#looping through the HTML table to scrape the data
    for row in rows:
        nick = row.find_all('td')[0].text
        team = [t.get('title') for t in row.find_all('td')[1].find_all('img')]
        maps = row.find_all('td')[2].text
        rounds = row.find_all('td')[3].text
        kddiff = row.find_all('td')[4].text
        killdeath = row.find_all('td')[5].text
        rating = row.find_all('td')[6].text
        print(team)
        print(nick)

Answered By: user510170

Python webscraper with Beautiful soup only returns one row of data

Question:

Answers: