Scraping href links in Python using BeautifulSoup
Question:
I am trying to scrape the links that are stored in the href values from this website: https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1
I tried the following code:
url = "https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1"
r = requests.get(url)
html_doc = r.content
soup = BeautifulSoup(html_doc, 'html.parser')
[tag['href'] for tag in soup.find_all('a',{'class':'css-11smhdq-FtContainer e1ym2d3s2'})]
But this just returns an empty list. Does anyone know how to fix this?
Answers:
The required data is loaded dynamically by JavaScript that’s why you are getting empty output because bs4 can’t render JS but they are using API and you can extract data from API url instead.
import requests
api_url= 'https://www.fotmob.com/api/teams?id=8676&timezone=Asia%2FDhaka&ccode3=BGD'
headers={
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
data=[]
res=requests.get(api_url,headers=headers)
#print(res)
for item in res.json()['fixtures']['allFixtures']['fixtures']:
url = item['pageUrl']
data.append({'url':'https://www.fotmob.com' + url})
print(data)
Output:
[{'url': 'https://www.fotmob.com/match/3915857/matchfacts/wycombe-wanderers-vs-burton-albion'}, {'url': 'https://www.fotmob.com/match/3915859/matchfacts/bolton-wanderers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3917827/matchfacts/northampton-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915875/matchfacts/wycombe-wanderers-vs-shrewsbury-town'}, {'url': 'https://www.fotmob.com/match/3915880/matchfacts/exeter-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915866/matchfacts/barnsley-vs-wycombe-wanderers'}, {'url':
'https://www.fotmob.com/match/3992753/matchfacts/wycombe-wanderers-vs-bristol-city'}, {'url': 'https://www.fotmob.com/match/3915922/matchfacts/wycombe-wanderers-vs-charlton-athletic'}, {'url': 'https://www.fotmob.com/match/3940648/matchfacts/wycombe-wanderers-vs-tottenham-hotspur-academy'}, {'url': 'https://www.fotmob.com/match/3915928/matchfacts/fleetwood-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915936/matchfacts/wycombe-wanderers-vs-accrington-stanley'}, {'url': 'https://www.fotmob.com/match/3915957/matchfacts/derby-county-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3940645/matchfacts/stevenage-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915945/matchfacts/sheffield-wednesday-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915987/matchfacts/wycombe-wanderers-vs-plymouth-argyle'}, {'url': 'https://www.fotmob.com/match/3915978/matchfacts/oxford-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915997/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3940637/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3916030/matchfacts/milton-keynes-dons-vs-wycombe-wanderers'}, {'url':
'https://www.fotmob.com/match/3916003/matchfacts/wycombe-wanderers-vs-cambridge-united'}, {'url': 'https://www.fotmob.com/match/3916015/matchfacts/wycombe-wanderers-vs-morecambe'}, {'url': 'https://www.fotmob.com/match/3915901/matchfacts/wycombe-wanderers-vs-port-vale'}, {'url': 'https://www.fotmob.com/match/4050422/matchfacts/wycombe-wanderers-vs-walsall'}, {'url': 'https://www.fotmob.com/match/3916041/matchfacts/forest-green-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916064/matchfacts/cheltenham-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916055/matchfacts/wycombe-wanderers-vs-portsmouth'}, {'url': 'https://www.fotmob.com/match/3916098/matchfacts/lincoln-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916093/matchfacts/wycombe-wanderers-vs-ipswich-town'}, {'url': 'https://www.fotmob.com/match/3916109/matchfacts/wycombe-wanderers-vs-bristol-rovers'}, {'url': 'https://www.fotmob.com/match/3916141/matchfacts/plymouth-argyle-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916154/matchfacts/peterborough-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916114/matchfacts/wycombe-wanderers-vs-oxford-united'}, {'url': 'https://www.fotmob.com/match/3916126/matchfacts/wycombe-wanderers-vs-sheffield-wednesday'}, {'url': 'https://www.fotmob.com/match/3916124/matchfacts/bristol-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916164/matchfacts/wycombe-wanderers-vs-fleetwood-town'}, {'url': 'https://www.fotmob.com/match/3916202/matchfacts/port-vale-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916186/matchfacts/wycombe-wanderers-vs-derby-county'}, {'url': 'https://www.fotmob.com/match/3916187/matchfacts/accrington-stanley-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916247/matchfacts/wycombe-wanderers-vs-bolton-wanderers'}, {'url': 'https://www.fotmob.com/match/3916223/matchfacts/shrewsbury-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916231/matchfacts/wycombe-wanderers-vs-exeter-city'}, {'url': 'https://www.fotmob.com/match/3916228/matchfacts/burton-albion-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916285/matchfacts/wycombe-wanderers-vs-barnsley'}, {'url': 'https://www.fotmob.com/match/3916279/matchfacts/charlton-athletic-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916314/matchfacts/wycombe-wanderers-vs-milton-keynes-dons'}, {'url': 'https://www.fotmob.com/match/3916321/matchfacts/ipswich-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916303/matchfacts/wycombe-wanderers-vs-forest-green-rovers'}, {'url': 'https://www.fotmob.com/match/3916358/matchfacts/morecambe-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916335/matchfacts/cambridge-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916349/matchfacts/wycombe-wanderers-vs-lincoln-city'}, {'url': 'https://www.fotmob.com/match/3916377/matchfacts/wycombe-wanderers-vs-cheltenham-town'}, {'url': 'https://www.fotmob.com/match/3916378/matchfacts/portsmouth-vs-wycombe-wanderers'}]
I am trying to scrape the links that are stored in the href values from this website: https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1
I tried the following code:
url = "https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1"
r = requests.get(url)
html_doc = r.content
soup = BeautifulSoup(html_doc, 'html.parser')
[tag['href'] for tag in soup.find_all('a',{'class':'css-11smhdq-FtContainer e1ym2d3s2'})]
But this just returns an empty list. Does anyone know how to fix this?
The required data is loaded dynamically by JavaScript that’s why you are getting empty output because bs4 can’t render JS but they are using API and you can extract data from API url instead.
import requests
api_url= 'https://www.fotmob.com/api/teams?id=8676&timezone=Asia%2FDhaka&ccode3=BGD'
headers={
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
data=[]
res=requests.get(api_url,headers=headers)
#print(res)
for item in res.json()['fixtures']['allFixtures']['fixtures']:
url = item['pageUrl']
data.append({'url':'https://www.fotmob.com' + url})
print(data)
Output:
[{'url': 'https://www.fotmob.com/match/3915857/matchfacts/wycombe-wanderers-vs-burton-albion'}, {'url': 'https://www.fotmob.com/match/3915859/matchfacts/bolton-wanderers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3917827/matchfacts/northampton-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915875/matchfacts/wycombe-wanderers-vs-shrewsbury-town'}, {'url': 'https://www.fotmob.com/match/3915880/matchfacts/exeter-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915866/matchfacts/barnsley-vs-wycombe-wanderers'}, {'url':
'https://www.fotmob.com/match/3992753/matchfacts/wycombe-wanderers-vs-bristol-city'}, {'url': 'https://www.fotmob.com/match/3915922/matchfacts/wycombe-wanderers-vs-charlton-athletic'}, {'url': 'https://www.fotmob.com/match/3940648/matchfacts/wycombe-wanderers-vs-tottenham-hotspur-academy'}, {'url': 'https://www.fotmob.com/match/3915928/matchfacts/fleetwood-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915936/matchfacts/wycombe-wanderers-vs-accrington-stanley'}, {'url': 'https://www.fotmob.com/match/3915957/matchfacts/derby-county-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3940645/matchfacts/stevenage-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915945/matchfacts/sheffield-wednesday-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915987/matchfacts/wycombe-wanderers-vs-plymouth-argyle'}, {'url': 'https://www.fotmob.com/match/3915978/matchfacts/oxford-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915997/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3940637/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3916030/matchfacts/milton-keynes-dons-vs-wycombe-wanderers'}, {'url':
'https://www.fotmob.com/match/3916003/matchfacts/wycombe-wanderers-vs-cambridge-united'}, {'url': 'https://www.fotmob.com/match/3916015/matchfacts/wycombe-wanderers-vs-morecambe'}, {'url': 'https://www.fotmob.com/match/3915901/matchfacts/wycombe-wanderers-vs-port-vale'}, {'url': 'https://www.fotmob.com/match/4050422/matchfacts/wycombe-wanderers-vs-walsall'}, {'url': 'https://www.fotmob.com/match/3916041/matchfacts/forest-green-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916064/matchfacts/cheltenham-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916055/matchfacts/wycombe-wanderers-vs-portsmouth'}, {'url': 'https://www.fotmob.com/match/3916098/matchfacts/lincoln-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916093/matchfacts/wycombe-wanderers-vs-ipswich-town'}, {'url': 'https://www.fotmob.com/match/3916109/matchfacts/wycombe-wanderers-vs-bristol-rovers'}, {'url': 'https://www.fotmob.com/match/3916141/matchfacts/plymouth-argyle-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916154/matchfacts/peterborough-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916114/matchfacts/wycombe-wanderers-vs-oxford-united'}, {'url': 'https://www.fotmob.com/match/3916126/matchfacts/wycombe-wanderers-vs-sheffield-wednesday'}, {'url': 'https://www.fotmob.com/match/3916124/matchfacts/bristol-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916164/matchfacts/wycombe-wanderers-vs-fleetwood-town'}, {'url': 'https://www.fotmob.com/match/3916202/matchfacts/port-vale-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916186/matchfacts/wycombe-wanderers-vs-derby-county'}, {'url': 'https://www.fotmob.com/match/3916187/matchfacts/accrington-stanley-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916247/matchfacts/wycombe-wanderers-vs-bolton-wanderers'}, {'url': 'https://www.fotmob.com/match/3916223/matchfacts/shrewsbury-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916231/matchfacts/wycombe-wanderers-vs-exeter-city'}, {'url': 'https://www.fotmob.com/match/3916228/matchfacts/burton-albion-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916285/matchfacts/wycombe-wanderers-vs-barnsley'}, {'url': 'https://www.fotmob.com/match/3916279/matchfacts/charlton-athletic-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916314/matchfacts/wycombe-wanderers-vs-milton-keynes-dons'}, {'url': 'https://www.fotmob.com/match/3916321/matchfacts/ipswich-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916303/matchfacts/wycombe-wanderers-vs-forest-green-rovers'}, {'url': 'https://www.fotmob.com/match/3916358/matchfacts/morecambe-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916335/matchfacts/cambridge-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916349/matchfacts/wycombe-wanderers-vs-lincoln-city'}, {'url': 'https://www.fotmob.com/match/3916377/matchfacts/wycombe-wanderers-vs-cheltenham-town'}, {'url': 'https://www.fotmob.com/match/3916378/matchfacts/portsmouth-vs-wycombe-wanderers'}]
