Remove duplicate 2D dict values

Question

I have the following list of dicts:

defaultdict(<class 'list'>,
            {'192.168.20.10/32': [{1: 'aaaaa11111n'},
                                  {2: 'bbbbb11111n'},
                                  {2: 'bbbbb11111n'},
                                  {2: 'bbbbb11111n'},
                                  {2: 'bbbbb11111n'}],
             '192.168.20.20/32': [{1: 'aaaaa11111n'},
                                  {2: 'aaaaa11111n'},
                                  {2: 'bbbbb11111n'},
                                  {2: 'bbbbb11111n'},
                                  {2: 'bbbbb11111n'},
                                  {2: 'bbbbb11111n'}]})

I would like to iterate over this list object and remove the duplicate list items.

The end result would look something like this:

defaultdict(<class 'list'>,
                {'192.168.20.10/32': [{1: 'aaaaa11111n'},
                                      {2: 'bbbbb11111n'}],
                 '192.168.20.20/32': [{1: 'aaaaa11111n'},
                                      {2: 'bbbbb11111n'}]})

Please note that I am using defaultdict from collections above:

from collections import defaultdict

Asked By: SeanCedric

||

Source

Answer 1

Maybe something like this (if resulting order does not matter):

adict = {
    '192.168.20.10/32': [
        {1: 'aaaaa11111n'},
        {2: 'bbbbb11111n'},
        {2: 'bbbbb11111n'},
        {2: 'bbbbb11111n'},
        {2: 'bbbbb11111n'}
    ],
    '192.168.20.20/32': [
        {1: 'aaaaa11111n'},
        {2: 'aaaaa11111n'},
        {2: 'bbbbb11111n'},
        {2: 'bbbbb11111n'},
        {2: 'bbbbb11111n'},
        {2: 'bbbbb11111n'}
    ]
}

for k, v in adict.items():
    adict[k] = list(map(dict, (set([tuple(obj.items()) for obj in v]))))
print(adict)

Answered By: funnydman

Answer 2

I’ve used the approach from https://stackoverflow.com/a/11092607/11574713.

Calling your dictionary first_dict, run the dicts through a set by converting them into a string (JSON), and then bring them back to dicts.

import json

for k in first_dict:
    first_dict[k] = [json.loads(j) for j in set([json.dumps(i) for i in first_dict[k]])]

defaultdict(list,
            {'192.168.20.10/32': [{'2': 'bbbbb11111n'},
              {'1': 'aaaaa11111n'}],
             '192.168.20.20/32': [{'2': 'aaaaa11111n'},
              {'2': 'bbbbb11111n'},
              {'1': 'aaaaa11111n'}]})

Answered By: 11574713

Remove duplicate 2D dict values

Question:

Answers: