BeautifulSoup get everything in a JSON list at specific element

Question

Day 2 of BS and JSON, just for a specific use case.

using requests and beautifulsoup4

found_json contains all the information I want, but I want a section of it.

page = requests.get(URL)
soup = bs(page.text, "html.parser")
found = soup.find("script", {"id": "__NEXT_DATA__"})
found_json = found.text   # json lives here

data example:

{"first_layer":{"second_layer":{"third_layer":{"id":"KYOU","liveData":{"....

I’m trying to get everything in that third layer and save it off somewhere. Is there any way to access that using BS, or should I look at the JSON library?

Asked By: maertsoi

||

Source

Answer 1

It’s better to use json

import json
... your code ...
found_json = json.loads(found.text)
data = found_json.get('first_layer').get('second_layer').get('third_layer')

Answered By: Rahul K P

BeautifulSoup get everything in a JSON list at specific element

Question:

Answers: