How to groupby two columns of a dataframe and convert other columns into dict with column header as key

• There is not a one-liner pandas argument that will provide a list of dicts in the shape you’re requesting.
• Use .groupby to select the groups
  • g is the tuple that represents the values used to groupby
  • d is the dataframe for the groupby value, g
• Use .iterrows to iterate through the rows of each group
  • Returns index represented by the first _, because it’s not needed
  • Returns data, from which the labels in groupby_list are dropped, then convert the remainder to a dict using .to_dict(), and append it to the list, att_list
  • After looping through all rows of the group, assign att_list as the value to group['attributes']
• Once each group has been iterated through, append the dict, group, to dict_list.
• dict_list can be converted back to a dataframe with the following:
  • df = pd.json_normalize(dict_list, 'attributes', meta=groupby_list)

dict_list = list()
groupby_list = ['id', 'id_2']
for g, d in df.groupby(groupby_list):
    group = dict(zip(groupby_list, g))
    att_list = list()
    for _, data in d.iterrows():
        data = data.drop(labels=groupby_list)
        att_list.append(data.to_dict())
    group['attributes'] = att_list
    dict_list.append(group)

dict_list:

[{'attributes': [{'allowance': 'No',
                  'name': 'Dave',
                  'salary': 120000,
                  'title': 'director'},
                 {'allowance': 'Yes',
                  'name': 'shaun',
                  'salary': 75000,
                  'title': 'developer'}],
  'id': 110,
  'id_2': 13.12},
 {'attributes': [{'allowance': 'No',
                  'name': 'Tom',
                  'salary': 100000,
                  'title': 'director'},
                 {'allowance': 'Yes',
                  'name': 'Sam',
                  'salary': 70000,
                  'title': 'developer'}],
  'id': 420,
  'id_2': 13.28}]