In Pandas, how to delete rows from a Data Frame based on another Data Frame?

Question

I have 2 Data Frames, one named USERS and another named EXCLUDE. Both of them have a field named "email".

Basically, I want to remove every row in USERS that has an email contained in EXCLUDE.

How can I do it?

Asked By: Vini

||

Source

Answer 1

You can use boolean indexing and condition with isin, inverting boolean Series is by ~:

import pandas as pd

USERS = pd.DataFrame({'email':['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']})
print (USERS)
     email
0  [email protected]
1  [email protected]
2  [email protected]
3  [email protected]
4  [email protected]

EXCLUDE = pd.DataFrame({'email':['[email protected]','[email protected]']})
print (EXCLUDE)
     email
0  [email protected]
1  [email protected]

print (USERS.email.isin(EXCLUDE.email))
0     True
1    False
2    False
3    False
4     True
Name: email, dtype: bool

print (~USERS.email.isin(EXCLUDE.email))
0    False
1     True
2     True
3     True
4    False
Name: email, dtype: bool

print (USERS[~USERS.email.isin(EXCLUDE.email)])
     email
1  [email protected]
2  [email protected]
3  [email protected]

Another solution with merge:

df = pd.merge(USERS, EXCLUDE, how='outer', indicator=True)
print (df)
     email     _merge
0  [email protected]       both
1  [email protected]  left_only
2  [email protected]  left_only
3  [email protected]  left_only
4  [email protected]       both

print (df.loc[df._merge == 'left_only', ['email']])
     email
1  [email protected]
2  [email protected]
3  [email protected]

Answered By: jezrael

Answer 2

You can also use inner join, take the indices or rows in USERS, that has email EXCLUDE, and then drop the them from the USERS. Following I use the @jezrael example to show this:

import pandas as pd
USERS = pd.DataFrame({'email': ['[email protected]',
                                '[email protected]',
                                '[email protected]',
                                '[email protected]',
                                '[email protected]']})

EXCLUDE = pd.DataFrame({'email':['[email protected]',
                                 '[email protected]']})

# rows in USERS and EXCLUDE with the same email
duplicates = pd.merge(USERS, EXCLUDE, how='inner',
                  left_on=['email'], right_on=['email'],
                  left_index=True)

# drop the indices from USERS
USERS = USERS.drop(duplicates.index)

This return:

USERS
    email
2   [email protected]
3   [email protected]
4   [email protected]

Answered By: Maryam Bahrami

Answer 3

My solution is just to find the elements is common, extract the shared key and then use that key to remove them from the original data:

emails2remove = pd.merge(USERS, EXCLUDE, how='inner', on=['email'])['email']
USERS = USERS[ ~USERS['email'].isin(emails2remove) ]

Answered By: Gabriel

Answer 4

Just to expand jezrael‘s answer, the same approach could be used in order to filter rows based on multiple columns.

USERS = pd.DataFrame({"email": ["[email protected]", "[email protected]", "[email protected]", 
                                "[email protected]", "[email protected]"],
                      "name": ["a", "s", "d", 
                               "f", "g"],
                      "nutrient_of_choice": ["pizza", "corn", "bread", 
                                             "coffee", "sausage"]})

print(USERS)    

     email name nutrient_of_choice
0  [email protected]    a              pizza
1  [email protected]    s               corn
2  [email protected]    d              bread
3  [email protected]    f             coffee
4  [email protected]    g            sausage

EXCLUDE = pd.DataFrame({"email":["[email protected]", "[email protected]"],
                        "name": ["a", "f"]})

print(EXCLUDE)

     email name
0  [email protected]    a
1  [email protected]    f

Now, suppose we would like to filter only rows with matching names and emails:

USERS = pd.merge(USERS, EXCLUDE, on=["email", "name"], how="outer", indicator=True)

print(USERS)

     email name nutrient_of_choice      _merge
0  [email protected]    a              pizza   left_only
1  [email protected]    s               corn   left_only
2  [email protected]    d              bread   left_only
3  [email protected]    f             coffee        both
4  [email protected]    g            sausage   left_only
5  [email protected]    a                NaN  right_only

USERS = USERS.loc[USERS["_merge"] == "left_only"].drop("_merge", axis=1)

print(USERS)

     email name nutrient_of_choice
0  [email protected]    a              pizza
1  [email protected]    s               corn
2  [email protected]    d              bread
4  [email protected]    g            sausage

Answered By: Kapara newbie

Answer 5

I know this post is old but it’s a fairly common question that needs an updated answer I would say.

I believe the best option is to use the loc operator

USERS[~USERS.loc[:,'EMAIL'].isin(EXCLUDE['EMAIL'])]

Answered By: Jason Perez

Answer 6

Another way is to use query:

USERS.query('email != @EXCLUDE["email"]')

@ is needed to access the other dataframe EXCLUDE.

Answered By: rachwa

In Pandas, how to delete rows from a Data Frame based on another Data Frame?

Question:

Answers: