Pandas filter a list of items present in a column

Question

I have a dataframe like as below

df = pd.DataFrame({'text': ["Hi how","I am fine","Ila say Hi"],
                   'tokens':[['Hi','how'],['I','am','fine'],['Ila','say','Hi']],
                    'labels':[['A','B'],['C','B','A'],['D','B','A']]})

I would like to do the below

a) Filter the df using tokens AND labels column

b) Filter based on the values Hi, Ila for tokens column

c) Filter based on the values A and D for labels column

So, I tried the below

df[((df['tokens']==['Hi'])&(df['tokens']==['Ila']))&((df['labels']==['A'])&(df['labels']==['D']))]

However, this doesn’t work. Since my column has values in list format, how do I filter them whether the list has only one item or multiple items?

I expect my output to be like as below

text          tokens             labels
Ila say Hi   [Ila, say, Hi]      [D, B, A]

Asked By: The Great

||

Source

Answer 1

You could try the following:

df.loc[
    df['tokens'].apply(lambda x: 'Hi' in x) &
    df['tokens'].apply(lambda x: 'Ila' in x) &
    df['labels'].apply(lambda x: 'A' in x) &
    df['labels'].apply(lambda x: 'D' in x) 
]

Output

         text          tokens     labels
2  Ila say Hi  [Ila, say, Hi]  [D, B, A]

You could also cast to string and use:

df.loc[
    df['tokens'].astype(str).str.contains('Hi') &
    df['tokens'].astype(str).str.contains('Ila') &
    df['labels'].astype(str).str.contains('A') &
    df['labels'].astype(str).str.contains('D') 
]

Answered By: Zach Flanders

Answer 2

Could you simply use a filter(lambda X: , df.columns) to get the columns you want, then just reindex the df?

Answered By: keynesiancross

Pandas filter a list of items present in a column

Question:

Answers: