Merging two list of dictionaries based on key

Question

dict1 = [{'id': 1.0, 'name': 'aa'},
 {'id': 4.0, 'name': 'bb'},
 {'id': 2.0, 'name': 'cc'}]

and

dict2 = [{'name': 'aa', 'dtype': 'StringType'},
 {'name': 'bb', 'dtype': 'StringType'},
 {'name': 'xx', 'dtype': 'StringType'},
 {'name': 'cc', 'dtype': 'StringType'}]

I would like to merge this two dictionaries based on their common key which is name.

I would like to get the following desired result.

merged_dict= [{'id': 1.0, 'name': 'aa', 'dtype': 'StringType'},
 {'id': 4.0, 'name': 'bb', 'dtype': 'StringType'},
 {'id': 2.0, 'name': 'cc', 'dtype': 'StringType'}]

I was trying to get this using the following for loop.

for i in dict1:
    for j in dict2:
         j.update(i)

Asked By: Hiwot

||

Source

Answer 1

To avoid quadratic complexity, better first create a real dictionary (yours are lists of dictionaries), then update:

tmp = {d['name']: d for d in dict2}

for d in dict1:
    d.update(tmp.get(d['name'], {}))

print(dict1)

Output:

[{'id': 1.0, 'name': 'aa', 'dtype': 'StringType'},
 {'id': 4.0, 'name': 'bb', 'dtype': 'StringType'},
 {'id': 2.0, 'name': 'cc', 'dtype': 'StringType'}]

Intermediate tmp:

{'aa': {'name': 'aa', 'dtype': 'StringType'},
 'bb': {'name': 'bb', 'dtype': 'StringType'},
 'xx': {'name': 'xx', 'dtype': 'StringType'},
 'cc': {'name': 'cc', 'dtype': 'StringType'}}

If you want a copy (rather that modifying dict1 in place):

tmp = {d['name']: d for d in dict2}
merged_dict = [d|tmp.get(d['name'], {}) for d in dict1]

Answered By: mozway

Answer 2

You can use pandas and try following:

import pandas as pd
df1 = pd.DataFrame(dict1)
df2 = pd.DataFrame(dict2)
res = df1.merge(df2, on=['name'])

The output:

    id name       dtype
0  1.0   aa  StringType
1  4.0   bb  StringType
2  2.0   cc  StringType

If you need a dictionary, you can convert merged result pd.DataFrame() to dict.

res.to_dict('records')

Final output is:

[
  {'id': 1.0, 'name': 'aa', 'dtype': 'StringType'}, 
  {'id': 4.0, 'name': 'bb', 'dtype': 'StringType'}, 
  {'id': 2.0, 'name': 'cc', 'dtype': 'StringType'}
]

Answered By: Ersel Er

Merging two list of dictionaries based on key

Question:

Answers: