Get lists in list with names of duplicate columns by values

Question

I have data frame:

import pandas as pd

data = [[101, 1, 2, 10, 3, 2, 3, 1], [5,5, 5, 5, 5, 5, 5, 5], [30, 3, 7, 14, 10, 7, 10, 2], [11, 2, 6, 15, 20, 6, 20, 11]] 

df = pd.DataFrame(data, columns = ['xen', 'sim', 'tab', 'sim', 'simm', 'box', 'simm', 'res'])

It looks like:

|   xen |   sim |   tab |   sim |   simm |   box |   simm |   res |
|------:|------:|------:|------:|-------:|------:|-------:|------:|
|   101 |     1 |     2 |    10 |      3 |     2 |      3 |     1 |
|     5 |     5 |     5 |     5 |      5 |     5 |      5 |     5 |
|    30 |     3 |     7 |    14 |     10 |     7 |     10 |     2 |
|    11 |     2 |     6 |    15 |     20 |     6 |     20 |    11 |

I need to get lists in list with names of duplicate columns by values (it can be duplicate by name or may not). For data frame above output should be like:

[["tab", "box"], ["simm", "simm"]]

Asked By: vasili111

||

Source

Answer 1

Try this:

l=df.T.reset_index().groupby(df.index.tolist())['index'].agg(list).loc[lambda x : x.str.len()>=2].values.tolist()
[['tab', 'box'], ['simm', 'simm']]

Answered By: BENY

Answer 2

Looks like you need to compare every pair of columns. So broadcast is an idea:

# extract the numpy array
values = df.to_numpy()

# compare columns by columns
rows, cols = np.where(np.triu((values[:,:,None] == values[:,None, :]).all(0), 1))

# output:
[df.columns[[r,c]].values for r,c in zip(rows,cols)]

Output:

[array(['tab', 'box'], dtype=object), array(['simm', 'simm'], dtype=object)]

Answered By: Quang Hoang

Answer 3

res = df.T.loc[df.T.duplicated(keep=False)]
pairs = res.sort_values(res.columns.tolist()).index
[ent.tolist() for ent in np.split(pairs,2)]
[['tab', 'box'], ['simm', 'simm']]

Answered By: sammywemmy

Answer 4

Here is a way:

s = df.apply(tuple,result_type='reduce')

s.reset_index().loc[s.duplicated(keep=False).tolist()].groupby(0)['index'].agg(list).tolist()

Output:

[['tab', 'box'], ['simm', 'simm']]

Answered By: rhug123

Get lists in list with names of duplicate columns by values

Question:

Answers: