Dropping duplicates in Pandas excluding one column

Question

This seems simple, but I can not find any information on it on the internet.

I have a dataframe like below:

City    State Zip           Date        Description       
Earlham IA    50072-1036    2014-10-10  Postmarket Assurance: Devices
Earlham IA    50072-1036    2014-10-10  Compliance: Devices
Madrid  IA    50156-1748    2014-09-10  Drug Quality Assurance

How can I eliminate duplicates that match 4 of 5 columns? The column not matching being Description.

The result would be

City    State Zip           Date        Description       
Earlham IA    50072-1036    2014-10-10  Postmarket Assurance: Devices
Madrid  IA    50156-1748    2014-09-10  Drug Quality Assurance

I found online that drop_duplicates with the subset parameter could work, but I am unsure of how I can apply it to multiple columns.

Asked By: Jstuff

||

Source

Answer 1

You’ve actually found the solution. For multiple columns, subset will be a list.

df.drop_duplicates(subset=['City', 'State', 'Zip', 'Date'])

Or, just by stating the column to be ignored:

df.drop_duplicates(subset=df.columns.difference(['Description']))

Answered By: ayhan

Dropping duplicates in Pandas excluding one column

Question:

Answers: