How to drop identical columns in Pandas dataframe if first x rows of values are identical?
Question:
I’m working with a large dataset (921600 rows, 23 columns) with the occasional duplicate column (different column names however). I would like to remove the columns with identical values. However, ‘df.T.drop_duplicates().T’ and similar solutions simply take too long as they presumably are checking all 921600 rows. Is it possible to remove columns if just the first x amount of rows have identical values?
E.g.: Identify that ‘channel2’ and ‘channel2-2’ are duplicate by comparing the first x (say 10) rows instead of inspecting all million rows.
channel1 channel2 channel3 channel2-b
0 47 46 27 46
1 84 28 28 28
2 72 79 68 79
... ... ... ... ...
999997 4729 1957 2986 1957
999998 9918 1513 2957 1513
999999 1001 5883 7577 5883
Answers:
Use DataFrame.duplicated with filter top values in DataFrame.head, filter rows by DataFrame.loc:
N = 2
df = df.loc[:, ~df.head(N).T.duplicated()]
I’m working with a large dataset (921600 rows, 23 columns) with the occasional duplicate column (different column names however). I would like to remove the columns with identical values. However, ‘df.T.drop_duplicates().T’ and similar solutions simply take too long as they presumably are checking all 921600 rows. Is it possible to remove columns if just the first x amount of rows have identical values?
E.g.: Identify that ‘channel2’ and ‘channel2-2’ are duplicate by comparing the first x (say 10) rows instead of inspecting all million rows.
channel1 channel2 channel3 channel2-b
0 47 46 27 46
1 84 28 28 28
2 72 79 68 79
... ... ... ... ...
999997 4729 1957 2986 1957
999998 9918 1513 2957 1513
999999 1001 5883 7577 5883
Use DataFrame.duplicated with filter top values in DataFrame.head, filter rows by DataFrame.loc:
N = 2
df = df.loc[:, ~df.head(N).T.duplicated()]