drop a dictionary with nan value

Question

I have the following dictionary:

my_dict = {'fields': ['id': 1.0,
  'name': 'aaa',
  'type': 'string'},
 {'id': 3.0,
  'name': 'eee',
  'type': 'string'},

 {'id': nan,
  'name': 'bbb',
  'type': 'string'},

 {'id': 4.0,
  'name': 'ccc',
  'type': 'string'},

 {'id': nan,
  'name': 'ddd',
  'type': 'string'}],

'type': 'struct'
}

From this dictionary, I would like to drop the dictionary with the id value nan value and would like to get the following.

my_updated_dict = {'fields': ['id': 1.0,
  'name': 'aaa',
  'type': 'string'},

 {'id': 3.0,
  'name': 'eee',
  'type': 'string'},

 {'id': 4.0,
  'name': 'ccc',
  'type': 'string'}],

'type': 'struct'
}

I was trying changing to data frame and dropping the id value with the nan value and changing to dictionary back but couldn’t get the intended result.

 my_updated_dict = pd.DataFrame(my_dict ).dropna().to_dict('list')

Asked By: Hiwot

||

Source

Answer 1

You can use update() to overwrite the value of the key, then you can try:

my_dict.update({'fields':[x for x in my_dict['fields'] if np.nan not in x.values()]})

Returning:

{'fields': [{'id': 1.0, 'name': 'aaa', 'type': 'string'},
  {'id': 3.0, 'name': 'eee', 'type': 'string'},
  {'id': 4.0, 'name': 'ccc', 'type': 'string'}],
 'type': 'struct'}

Answered By: Celius Stingher

Answer 2

I do not know why would you need pandas for that if u can simply do:

my_dict["fields"] = [i for i in my_dict["fields"] if not np.isnan(i["id"])]

** UPDATE **

if you really do need for some reason to use pandas, you may try this constructiion:

my_dict["fields"] = pd.Series(my_dict["fields"]).apply(pd.Series).dropna().to_dict(orient="records")

though I do not see any advantages over simple list comprehension, except may be on big volume of information.

Answered By: strawdog

Answer 3

Considering the dictionary json

import numpy as np

json = {'fields': [{'id': 1.0, 'name': 'aaa', 'type': 'string'},
                     {'id': 3.0, 'name': 'eee', 'type': 'string'},
                        {'id': np.nan, 'name': 'bbb', 'type': 'string'},
                            {'id': 4.0, 'name': 'ccc', 'type': 'string'},
                                {'id': np.nan, 'name': 'ddd', 'type': 'string'}],
                                    'type': 'struct'}

In order to remove the parts where id is np.nan, one can use a list comprehension with numpy.isnan as follows

json['fields'] = [x for x in json['fields'] if not np.isnan(x['id'])]

[Out]:

{'fields': [{'id': 1.0, 'name': 'aaa', 'type': 'string'},
            {'id': 3.0, 'name': 'eee', 'type': 'string'},
            {'id': 4.0, 'name': 'ccc', 'type': 'string'}],
 'type': 'struct'}

Answered By: Gonçalo Peres

drop a dictionary with nan value

Question:

Answers: