Pulling paginated json data with python / loop

Question

I pulled json data with api. Response example as below.

{
  "page": 1,
  "pages": 5,
  "limit": 100,
  "count": 466,
  "records": [
    {
      "epoch": "9b4WLoXXYgga9mGRLGsWKu",
      "last_height": 5402,
      "last_time": "2021-02-22T01:02:43.011692Z",
      "expected_blocks": 432,
      "produced_blocks": 432,
    }
  ]
}

But page is limited with 100 rows. Parameters query is available for pages.
Below is python code I use:

import requests
import json
import pandas as pd

response = requests.get(url, params = {'page': 1})
json_data = json.loads(response.text)
df = pd.DataFrame(json_data['records']) 
print(df)

How can I get all pages with looping?

Thanks

Asked By: Alper

||

Source

Answer 1

You can try this and get all your data in dataframe by looping.

import requests
import json
import pandas as pd

response = requests.get(url, params = {'page': 1}) #---> remember to keep your url 
records = []
for page_number in range(2, response.json().get("pages")+1):
    response = requests.get(url, params = {'page': page_number})
    records += response.json().get('records')
df = pd.DataFrame(records) 
print(df)

Answered By: Divya Prakash

Pulling paginated json data with python / loop

Question:

Answers: