Accessing tabbed elements with BeautifulSoup

Question

I want to scrape all the currency tabs under Currency pairs on this site: https://markets.businessinsider.com/currencies. However I can only get the currency pairs that are active by default when I load the page which is US-Dollar. How can I access the rest of the items with Beautifulsoup?

Using the code below I get all the US-dollar pairs, but none of the other pairs.

from bs4 import BeautifulSoup

response = requests.get("https://markets.businessinsider.com/currencies")
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser")

currencyTags = soup.find_all("td", class_="table__td bold")
print(currencyTags)```

Asked By: Siggerud

||

Source

Answer 1

You can’t with bs4 (unless you’re obtaining the html with something that supports JavaScript and automated interaction with the page, like selenium), but you can request the data from the same API that the site fetches it from^{(view screenshot from network logs)}.

For example, for Japanese Yen:

import request
import pandas as pd

cur_tab = 'JPY' # 'MostViewed' # 'USD' # 'EUR' #
api_endpt = f'https://markets.businessinsider.com/ajax/ExchangeRate_PopularListData' 
api_data = requests.get(f'{api_endpt}?currency={cur_tab}').json()
df = pd.DataFrame(api_data['ExchangeRates'])

Or to get all of the tabs:

import request
import pandas as pd

cur_tabs = ['MostViewed', 'USD', 'EUR', 'JPY']
api_endpt = 'https://markets.businessinsider.com/ajax/ExchangeRate_PopularListData'

dfList = [pd.DataFrame(
    requests.get(f'{api_endpt}?currency={cur_tab}').json()['ExchangeRates']
) for cur_tab in cur_tabs] 
df = pd.concat(dfList, keys=cur_tabs)

Answered By: Driftr95

Accessing tabbed elements with BeautifulSoup

Question:

Answers: