Pandas groupby and make set of items

Question

I am using pandas groupby and want to apply the function to make a set from the items in the group.

The following results in TypeError: 'type' object is not iterable:

df = df.groupby('col1')['col2'].agg({'size': len, 'set': set})

But the following works:

def to_set(x):
    return set(x)
    
df = df.groupby('col1')['col2'].agg({'size': len, 'set': to_set})

In my understanding the two expression are similar, what is the reason why the first does not work?

Asked By: hangc

||

Answer 1

Update

As late as pandas version 0.22, this is an issue.
As of pandas version 1.1.2, this is not an issue. Aggregating set, doesn’t result in TypeError: 'type' object is not iterable.
- Not certain when the functionality was updated.

It’s because set is of type type whereas to_set is of type function:

type(set)
<class 'type'>

def to_set(x):
    return set(x)

type(to_set)

<class 'function'>

According to the docs, .agg() expects:

arg : function or dict

Function to use for aggregating groups.

If a function, must either work when passed a DataFrame or when passed to
DataFrame.apply.

If passed a dict, the keys must be DataFrame column
names.

Accepted Combinations are:

string cythonized function name

function

list of functions

dict of columns -> functions

nested dict of names -> dicts of functions

Answered By: Stefan

Answer 2

Try using:

df = df.groupby('col1')['col2'].agg({'size': len, 'set': lambda x: set(x)})

Works for me.

Answer 3

Update for newer versions of Pandas if you get the following error

SpecificationError: nested renamer is not supported

df = df.groupby('col1')['col2'].agg(size= len, set= lambda x: set(x))

Answered By: kindjacket

Answer 4

Update for Pandas version 1.3.3 if using .agg({'set': set}) produces the following error:

TypeError: Unable to infer the type of the field set

This persists if simply using the previously suggested solution of .agg({'set': lambda x: set(x)})

The reason for this is that set does not fulfil is_list_like in _aggregate (detailed explanation here, courtesy of @EdChum)

A solution is therefore to coerce it to a list using:

.agg({'set': lambda x: list(set(x))})

Answered By: tomlincr