Python: Iterate and modify a complex dict including lists

Question

I need to iterate through a complex dictionary and modify it according to several conditions.

This is a very basic example of the dict

a = {
   'x' : ["x1", "x2", 'x3'],
   'y' : [
       {'y1' : 1},
       {'y2' : 2.0},
       {'y3' : True},
       {'y4' : 99},
   ],
   'z' : {
      'x' : ["a1", "a2", 'a3'],
      'y' : [
           {'y1' : 66},
           {'y2' : False},
           {'y3' : 3},
           {'y4' : 4.3},
       ]
   },
   'y3' : "Delete Me"
}

Îterating through it, is also no problem like:

 pTypes = [str, bool, int, float]
 def parse(d, c):
     t = type(d)
     if t == dict:
         for k, v in d.items():
             parse(v, c+[k])
     elif t == list:
         for i, p in enumerate(d):
             parse(p,c+[i])
     elif t in pTypes:
         print(c,'=>',d)
     else:
         print('Error: ',c,d,t)


 parse(a,[])

My problem is, that I need to delete all items which have a key == ‘y3’ or a value > 50

Additionally all strings need to be enclosed by ‘-‘ like ‘x1’ => ‘-x1-‘

Just modifing the data does not work (is not persistent)

...
elif t in pTypes:
   if t == str:
       d = '-'+d+'-'
...

And I have no idea how to remove items while iterating the dict itself.

Since the dict can be huge, i would prefer not to make a "reduced" copy of the original.

Can anybody help?

Asked By: JustMe

||

Source

Answer 1

The exact expected output is unclear, but since you are iterating a nested structure, better generate a new object.

Keeping your original recursive logic, you should add a return at each step to propagate your objects:

def parse(d):
    if isinstance(d, dict):
        return {k: parse(v) for k, v in d.items()
                if k != 'y3' and not
                (isinstance(v, (int, float)) and v>50)}
    elif isinstance(d, list):
        return [parse(x) for x in d]
    elif isinstance(d, str):
        return f'-{d}-'
    else:
        return d

out = parse(a)

Output:

{'x': ['-x1-', '-x2-', '-x3-'],
 'y': [{'y1': 1}, {'y2': 2.0}, {}, {}],
 'z': {'x': ['-a1-', '-a2-', '-a3-'],
       'y': [{}, {'y2': False}, {}, {'y4': 4.3}]}}

Answered By: mozway

Answer 2

Mozway’s answer is entirely satisfying if you do not care about empty dictionaries replacing the ones that did not fill your conditions. You did not provide a lot of details, but I can imagine cases where you could care. As an example, if your original dict already contained empty dicts, then you would not be able to distinguish between those original empty dicts (maybe missing data ?) and the created empty dicts (deleted data).

To fix this, you could "mark" emptied dicts by replacing them by something else, in my example, I use the string 'This marks an emptied dict'. To make sure I’m dealing with an emptied dict, I use copies, modify them, and then compare them to the original. The last two lines delete the marked items, but you could want to keep them as well.

import copy

def parse(anything) :
    c = copy.copy(anything)
    
    if isinstance(anything, dict) :
        for key, value in anything.items() :
            if key == 'y3' or (isinstance(value, (int, float)) and value > 50) :
                del c[key]
            else :
                c[key] = parse(value)
        if len(c) == 0 and len(anything) != 0 :
            return 'This marks an emptied dict'
    elif isinstance(anything, list) :
        c = [parse(item) for item in anything]
    elif isinstance(anything, str) :
        c = '-' + c + '-'

    if isinstance(c, list) :
        c = [item for item in c if not (isinstance(item, str) and item == 'This marks an emptied dict')]
    
    return c

parse(a)

Output :

{'x': ['-x1-', '-x2-', '-x3-'],
 'y': [{'y1': 1}, {'y2': 2.0}],
 'z': {'x': ['-a1-', '-a2-', '-a3-'], 'y': [{'y2': False}, {'y4': 4.3}]}}

Answered By: chatours

Python: Iterate and modify a complex dict including lists

Question:

Answers: