How to handle missing JSON nested keys from an API response in python?

Question

Here is the JSON response I get from an API request:

{  
    "associates": [
        {
            "name":"DOE",
            "fname":"John",
            "direct_shares":50,
            "direct_shares_details":{
               "shares_PP":25,
               "shares_NP":25
            },
            "indirect_shares":50,
            "indirect_shares_details": {
                  "first_type": {
                      "shares_PP": 25,
                      "shares_NP": 0
                  },
                  "second_type": {
                      "shares_PP": 25,
                      "shares_NP": 0
                  }
            }
        } 
    ]
}

However, in some occasions, some values will be equal to None. In that case, I handle it in my function for all the values that I know will be integers. But it doesn’t work in this scenario for the nested keys inside indirect_shares_details:

{  
    "associates": [
        {
            "name":"DOE",
            "fname":"John",
            "direct_shares":50,
            "direct_shares_details":{
               "shares_PP":25,
               "shares_NP":25
            },
            "indirect_shares":None,
            "indirect_shares_details": None
            }
        } 
    ]
}

So when I run my function to get the API values and put them in a custom dict, I get an error because the keys are simply inexistant in the response.

def get_shares_data(response):
    associate_from_api = []
        for i in response["associates"]:
            associate_data = {
            "PM_shares":  round(company["Shares"], 2),
            "full_name": i["name"] + " " + ["fname"]
            "details": {
                    "shares_in_PM":    i["direct_shares"],
                    "shares_PP_in_PM": i["direct_shares_details"]["shares_PP"],
                    "shares_NP_in_PM": i["direct_shares_details"]["shares_NP"],
                    "shares_directe":  i["indirect_shares"],
                    "shares_indir_PP_1": i["indirect_shares_details"]["first_type"]["shares_PP"],
                    "shares_indir_NP_1": i["indirect_shares_details"]["first_type"]["shares_NP"],
                    "shares_indir_PP_2": i["indirect_shares_details"]["second_type"]["shares_PP"],
                    "shares_indir_NP_2": i["indirect_shares_details"]["second_type"]["shares_NP"],
                }
            }
            for key,value in associate_data["details"].items():
                if value != None:
                    associate_data["details"][key] = value * associate_data["PM_shares"] / 100
                else:
                    associate_data["calculs"][key] = 0.0
            associate_from_api.append(associate_data)
    return associate_from_api

I’ve tried conditioning the access of the nested keys only if the parent key wasn’t equal to None but I ended up declaring 3 different dictionaries inside if/else conditions and it turned into a mess, is there an efficient way to achieve this?

Asked By: user16779293

||

Source

Answer 1

You can try accessing the values using dict.get('key') instead of accessing them directly, as in dict['key'].

Using the first approach, you will get None instead of KeyError if the key is not there.

EDIT: tested using the dictionary from the question:

Answered By: Victor Valente

Answer 2

You can try pydantic

Install pydantic

pip install pydantic

# OR

conda install pydantic -c conda-forge

Define some models based on your response structure

from pydantic import BaseModel
from typing import List, Optional


# There are some common fields in your json response.
# So you can put them together.
class ShareDetail(BaseModel):
    shares_PP: int
    shares_NP: int


class IndirectSharesDetails(BaseModel):
    first_type: ShareDetail
    second_type: ShareDetail


class Associate(BaseModel):
    name: str
    fname: str
    direct_shares: int
    direct_shares_details: ShareDetail
    indirect_shares: int = 0  # Sets a default value for this field.
    indirect_shares_details: Optional[IndirectSharesDetails] = None


class ResponseModel(BaseModel):
    associates: List[Associate]

use ResponseModel.parse_xxx functions to parse response.

Here I use parse_file funtion, you can also use parse_json function

See: https://pydantic-docs.helpmanual.io/usage/models/#helper-functions

def main():
    res = ResponseModel.parse_file("./NullResponse.json",
                                   content_type="application/json")

    print(res.dict())


if __name__ == "__main__":
    main()

Then the response can be successfully parsed. And it automatically validates the input.

Answered By: 九月家的敖烈

How to handle missing JSON nested keys from an API response in python?

Question:

Answers: