How to automatically parse JSON and get all key value pairs in a variable?

Question

I’m trying to dynamically create a database without hard coding all of the API response values for columns and rows so I want to learn how to automatically parse through JSON and put all the keys/values in a variable so I can sort through and put them in the db.

Lets say I had some JSON like this:

(modified snippet from destiny 2 API response indented by 5)

{
     "Response": {
          "profile": {
               "data": {
                    "userInfo": {
                         "crossSaveOverride": 1,
                         "applicableMembershipTypes": [
                              3,
                              1
                         ],
                         "isPublic": true,
                         "membershipType": 1,
                         "membershipId": "123",      
                         "displayName": "Name1",
                         "bungieGlobalDisplayName": "name again",
                         "bungieGlobalDisplayNameCode": 0000
                    },
                    "dateLastPlayed": "2021-6-18T02:33:01Z",        
                    "versionsOwned": 127,
                    "characterIds": [
                         "id1",
                         "id2",
                         "id3"
                    ],
                    "seasonHashes": [
                         3243434344,
                         3423434244,
                         3432342443,
                         3434243244
                    ],
                    "currentSeasonHash": 3434243244,
                    "currentSeasonRewardPowerCap": 1330
               },
               "privacy": 1
          }
     },
     "ErrorCode": 1,
     "ThrottleSeconds": 0,
     "ErrorStatus": "Success",
     "Message": "Ok",
     "MessageData": {}
}

I want to automatically parse through and get all of the key/values minus the error code area so everything under "Response". It would all go into the database for example:

displayName	isPublic
Name1	True
Name2	False

I know how to parse normally or loop through but only to grab values like:

displayName = Response["Response"]["profile"]["data"]["userInfo"]["displayName"]

How does one grab all keys and values and then automatically store in a variable from the lowest level? Also, how would one exclude keys if they are not needed?

EDIT: Adding Clarification

I learned that this JSON response type is a dict and I can use Response.keys() and Response.values() to get the keys and values.

What I am asking is, from Response["Response"], how to get all of the keys and values down to the bottom level and exclude the ones I don’t need.

For example if I do:

r = Response["Response"] 
for key in r.keys():
    print(key)

I’m only going to get profile which is what I don’t need.

I would have to do:

r = Response["Response"]["profile"]["data"]["userInfo"] 
for key in r.keys():
    print(key)

which would return

crossSaveOverride
applicableMembershipTypes
isPublic
membershipType
membershipId
displayName
bungieGlobalDisplayName
bungieGlobalDisplayNameCode

Which is what I need but I do not want to manually define
["Response"]["profile"]["data"]["userInfo"]or similar for every response. I want to grab everything automatically while excluding the items I don’t need.

Asked By: dekubaka

||

Source

Answer 1

Dmitry Torba over at Get all keys of a nested dictionary posted a function that does what you want.

def recursive_items(dictionary):
    for key, value in dictionary.items():
        if type(value) is dict:
            yield from recursive_items(value)
        else:
            yield (key, value)

a = {'a': {1: {1: 2, 3: 4}, 2: {5: 6}}}

for key, value in recursive_items(a):
    print(key, value)

Answered By: iScripters

Answer 2

I have tried this approach, what if there is list type in given dictionary!

CallPretty.py

def JsonPrint(response, response_Keys):
    print()
    print(response_Keys)
    for i in range(len(response_Keys)):

        if type(response[response_Keys[i]]) == list:
            print()
            
            for j in range(len(response[response_Keys[i]])):
                if type(response[response_Keys[i]][j]) == dict:
                    JsonPrint(response[response_Keys[i]][j], list(response[response_Keys[i]][j].keys()))

        elif type(response[response_Keys[i]]) == dict:
            JsonPrint(response[response_Keys[i]], list(response[response_Keys[i]].keys()))

        else:
            print(response_Keys[i], ':')
            for j in range(len(response[response_Keys[i]].split('n'))):
                print('t', response[response_Keys[i]].split('n')[j])

Answered By: Vicky Kumar

How to automatically parse JSON and get all key value pairs in a variable?

Question:

EDIT: Adding Clarification

Answers: