Add blank value for key if does not exist when appending dictionaries to list

Question

I have the following code which uses an API to download data from a website.

url = "https://gtr.ukri.org/project"
params = {"page": 11,
          "fetchSize": 100}
projects = requests.get(url,
                        params=params,
                        headers={"accept": "application/json"}).json()['projectsBean']['project']

for project in projects:
    list_projects.append({'ref': project['grantReference'], 'funder': project['fund']['funder']['name'],
                          'title': project['title'], 'category': project['grantCategory'],
                          'abstract': project['abstractText'], 'funding': project['fund']['valuePounds'],
                          'start': project['fund']['start'], 'end': project['fund']['end']})

This is working fine, however sometimes there is no value for a key when iterating through all the pages in the website (the key will always exist).

When this happens, it errors as a value is not there.

How do I append a dictionary to a list and just leave any values for any key blank if it does not exist, rather than erroring?

Many thanks

Asked By: Nicholas

||

Source

Answer 1

To have a default value when no value exists in your dictionary you can do it like this

project.get('grantReference', 'no-ref')

the string 'no-ref' is a default value if the project dict doesn’t contain any key named grantReference.

the above is only an example to do get a value of dict with key without error.

but however, on your question

This is working fine, however sometimes there is no value for a key when iterating through all the pages in the website (the key will always exist).

you doesn’t mention the exact code line where the error come from so I can not help to create a complete example code.

For nested dictionary you can use the method like this

project.get('fund', {}).get('start', 0)

the first key 'fund' will return value that you expecting or will default to be an empty dictionary. so the second key 'start' will get value with key 'start' from the dictionary you expecting or from the default empty dictionary we provide. (and we provide default value for key 'start' which is the 0)

Answered By: Fran N.J.

Answer 2

Another way of doing this is to override the __missing__ operator on dict and use an object_hook on json.loads on your response string, which will work recursviely.

import json

class GetItemNone(dict):

    def __missing__(self, key):
        return None


example = """{
    "foo": {
        "bar": 1
    }
}"""


d = json.loads(example, object_hook=GetItemNone)

Examples:

d["foo"]["bar"] -> 1
d["foo"]["foo"] -> None
d["bar"] -> None

Answered By: Tom McLean

Answer 3

I think this approach is what you are looking for:

from functools import reduce

def get_object(obj, key):
    if isinstance(obj, dict):
        if key in list(obj.keys()):
            return obj[key]
        else: 
            return ""
    else:
        return ""

projects =[{'grantReference': 'my_ref', 'fund':{'funder':{'name': "my_name"},
                                   'valuePounds': 25}}, 
           {'grantReference': 'my_ref', 'fund':{'start': "today",
                                   'end': "tomorrow"}}]
    
list_projects = []
for project in projects:
    list_projects.append({'ref': reduce(get_object, ['grantReference'], project),
                          'funder': reduce(get_object,['fund', 'funder', 'name'], project),
                          'title': reduce(get_object, ['title'], project), 
                          'category': reduce(get_object,  ['grantCategory'], project),
                          'abstract': reduce(get_object, ['abstractText'], project), 
                          'funding': reduce(get_object, ['fund', 'valuePounds'], project),
                          'start': reduce(get_object, ['fund', 'start'], project), 
                          'end': reduce(get_object, ['fund', 'end'], project)})



list_projects

output:

[{'ref': 'my_ref',
  'funder': 'my_name',
  'title': '',
  'category': '',
  'abstract': '',
  'funding': 25,
  'start': '',
  'end': ''},
 {'ref': 'my_ref',
  'funder': '',
  'title': '',
  'category': '',
  'abstract': '',
  'funding': '',
  'start': 'today',
  'end': 'tomorrow'}]

Answered By: Lucas M. Uriarte

Add blank value for key if does not exist when appending dictionaries to list

Question:

Answers: