How do I get a list of all the duplicate items using pandas in python?

Using an element-wise logical or and setting the take_last argument of the pandas duplicated method to both True and False you can obtain a set from your dataframe that includes all of the duplicates.

df_bigdata_duplicates = 
    df_bigdata[df_bigdata.duplicated(cols='ID', take_last=False) |
               df_bigdata.duplicated(cols='ID', take_last=True)
              ]

With Pandas version 0.17, you can set ‘keep = False’ in the duplicated function to get all the duplicate items.

In [1]: import pandas as pd

In [2]: df = pd.DataFrame(['a','b','c','d','a','b'])

In [3]: df
Out[3]: 
       0
    0  a
    1  b
    2  c
    3  d
    4  a
    5  b

In [4]: df[df.duplicated(keep=False)]
Out[4]: 
       0
    0  a
    1  b
    4  a
    5  b

df[df.duplicated(['ID'], keep=False)]

it’ll return all duplicated rows back to you.

According to documentation:

keep : {‘first’, ‘last’, False}, default ‘first’

• ‘first’ : Mark duplicates as True except for the first occurrence.
• ‘last’ : Mark duplicates as True except for the last occurrence.
• False : Mark all duplicates as True.

df[df['ID'].duplicated() == True]

This worked for me

This may not be a solution to the question, but to illustrate examples:

import pandas as pd

df = pd.DataFrame({
    'A': [1,1,3,4],
    'B': [2,2,5,6],
    'C': [3,4,7,6],
})

print(df)
df.duplicated(keep=False)
df.duplicated(['A','B'], keep=False)

The outputs:

   A  B  C
0  1  2  3
1  1  2  4
2  3  5  7
3  4  6  6

0    False
1    False
2    False
3    False
dtype: bool

0     True
1     True
2    False
3    False
dtype: bool

As I am unable to comment, hence posting as a separate answer

To find duplicates on the basis of more than one column, mention every column name
as below, and it will return you all the duplicated rows set:

df[df[['product_uid', 'product_title', 'user']].duplicated() == True]

Alternatively,

df[df[['product_uid', 'product_title', 'user']].duplicated()]

You could use:

df[df.duplicated(['ID'])==True].sort_values('ID')

duplicated rows and their index loc # for all column values

def dup_rows_index(df):
  dup = df[df.duplicated()]
  print('Duplicated index loc:',dup[dup == True ].index.tolist())
  return dup

sort("ID") does not seem to be working now, seems deprecated as per sort doc, so use sort_values("ID") instead to sort after duplicate filter, as following:

df[df.ID.duplicated(keep=False)].sort_values("ID")

For my database .duplicated(keep=False) did not work until the column was sorted.

data.sort_values(by=['Order ID'], inplace=True)
df = data[data['Order ID'].duplicated(keep=False)]

Inspired by the solutions above, you can further sort values so that you can look at the records that are duplicated sorted:

df[df.duplicated(['ID'], keep=False)].sort_values(by='ID')

This code gives you a data frame indicating if a row has any repetition in the data frame:

df2 = df1.duplicated()

This code eliminates the duplications and keeps only one instance:

df3 = df1.drop_duplicates(keep="first")

df3 will be a data frame consisting from the unique items (rows).

TL;DR

This worked for me:

dups = [i for i, v in df["Col1"].value_counts().iteritems() if v > 1]
dups

Output:

[501, 505]

To list duplicate rows:

fltr = df["Col1"].isin(dups)  # Filter
df[fltr]

Output:

   Col1  Col2
0   501     D
1   501     H
2   505     E
3   501     E
4   505     M

Explanation:

Taking value_counts() of a column, say Col1, returns a Series with:

1. Data of column Col1 as the Series index.
2. Count of values in Col1 as the Series values.

For example value_counts() on below DataFrame:

df["Col1"].value_counts()

Outputs below Series:

501    3
505    2
502    1
503    1
Name: Col1, dtype: int64

Now using iteritems() we can access both index and values of a Series object:

dups = [i for i, v in df["Col1"].value_counts().iteritems() if v > 1]
dups

Output:

[501, 505]

Now use the duplicate values captured as filter on original DataFrame.

fltr = df["Col1"].isin(dups)  # Filter
df[fltr]

   Col1  Col2
0   501     D
1   501     H
2   505     E
3   501     E
4   505     M