Shuffle DataFrame rows
Question:
I have the following DataFrame:
Col1 Col2 Col3 Type
0 1 2 3 1
1 4 5 6 1
...
20 7 8 9 2
21 10 11 12 2
...
45 13 14 15 3
46 16 17 18 3
...
The DataFrame is read from a CSV file. All rows which have Type
1 are on top, followed by the rows with Type
2, followed by the rows with Type
3, etc.
I would like to shuffle the order of the DataFrame’s rows so that all Type
‘s are mixed. A possible result could be:
Col1 Col2 Col3 Type
0 7 8 9 2
1 13 14 15 3
...
20 1 2 3 1
21 10 11 12 2
...
45 4 5 6 1
46 16 17 18 3
...
How can I achieve this?
Answers:
You can shuffle the rows of a data frame by indexing with a shuffled index. For this, you can eg use np.random.permutation
(but np.random.choice
is also a possibility):
In [12]: df = pd.read_csv(StringIO(s), sep="s+")
In [13]: df
Out[13]:
Col1 Col2 Col3 Type
0 1 2 3 1
1 4 5 6 1
20 7 8 9 2
21 10 11 12 2
45 13 14 15 3
46 16 17 18 3
In [14]: df.iloc[np.random.permutation(len(df))]
Out[14]:
Col1 Col2 Col3 Type
46 16 17 18 3
45 13 14 15 3
20 7 8 9 2
0 1 2 3 1
1 4 5 6 1
21 10 11 12 2
If you want to keep the index numbered from 1, 2, .., n as in your example, you can simply reset the index: df_shuffled.reset_index(drop=True)
The idiomatic way to do this with Pandas is to use the .sample
method of your data frame to sample all rows without replacement:
df.sample(frac=1)
The frac
keyword argument specifies the fraction of rows to return in the random sample, so frac=1
means to return all rows (in random order).
Note:
If you wish to shuffle your dataframe in-place and reset the index, you could do e.g.
df = df.sample(frac=1).reset_index(drop=True)
Here, specifying drop=True
prevents .reset_index
from creating a column containing the old index entries.
Follow-up note: Although it may not look like the above operation is in-place, python/pandas is smart enough not to do another malloc for the shuffled object. That is, even though the reference object has changed (by which I mean id(df_old)
is not the same as id(df_new)
), the underlying C object is still the same. To show that this is indeed the case, you could run a simple memory profiler:
$ python3 -m memory_profiler .test.py
Filename: .test.py
Line # Mem usage Increment Line Contents
================================================
5 68.5 MiB 68.5 MiB @profile
6 def shuffle():
7 847.8 MiB 779.3 MiB df = pd.DataFrame(np.random.randn(100, 1000000))
8 847.9 MiB 0.1 MiB df = df.sample(frac=1).reset_index(drop=True)
You can simply use sklearn
for this
from sklearn.utils import shuffle
df = shuffle(df)
TL;DR: np.random.shuffle(ndarray)
can do the job.
So, in your case
np.random.shuffle(DataFrame.values)
DataFrame
, under the hood, uses NumPy ndarray as a data holder. (You can check from DataFrame source code)
So if you use np.random.shuffle()
, it would shuffle the array along the first axis of a multi-dimensional array. But the index of the DataFrame
remains unshuffled.
Though, there are some points to consider.
- function returns none. In case you want to keep a copy of the original object, you have to do so before you pass to the function.
sklearn.utils.shuffle()
, as user tj89 suggested, can designate random_state
along with another option to control output. You may want that for dev purposes.
sklearn.utils.shuffle()
is faster. But WILL SHUFFLE the axis info(index, column) of the DataFrame
along with the ndarray
it contains.
Benchmark result
between sklearn.utils.shuffle()
and np.random.shuffle()
.
ndarray
nd = sklearn.utils.shuffle(nd)
0.10793248389381915 sec. 8x faster
np.random.shuffle(nd)
0.8897626010002568 sec
DataFrame
df = sklearn.utils.shuffle(df)
0.3183923360193148 sec. 3x faster
np.random.shuffle(df.values)
0.9357550159329548 sec
Conclusion: If it is okay to axis info(index, column) to be shuffled along with ndarray, use sklearn.utils.shuffle()
. Otherwise, use np.random.shuffle()
used code
import timeit
setup = '''
import numpy as np
import pandas as pd
import sklearn
nd = np.random.random((1000, 100))
df = pd.DataFrame(nd)
'''
timeit.timeit('nd = sklearn.utils.shuffle(nd)', setup=setup, number=1000)
timeit.timeit('np.random.shuffle(nd)', setup=setup, number=1000)
timeit.timeit('df = sklearn.utils.shuffle(df)', setup=setup, number=1000)
timeit.timeit('np.random.shuffle(df.values)', setup=setup, number=1000)
shuffle the pandas data frame by taking a sample array in this case index and randomize its order then set the array as an index of data frame. Now sort the data frame according to index. Here goes your shuffled dataframe
import random
df = pd.DataFrame({"a":[1,2,3,4],"b":[5,6,7,8]})
index = [i for i in range(df.shape[0])]
random.shuffle(index)
df.set_index([index]).sort_index()
output
a b
0 2 6
1 1 5
2 3 7
3 4 8
Insert you data frame in the place of mine in above code .
Here is another way to do this:
df_shuffled = df.reindex(np.random.permutation(df.index))
(I don’t have enough reputation to comment this on the top post, so I hope someone else can do that for me.) There was a concern raised that the first method:
df.sample(frac=1)
It makes a deep copy or just changed the dataframe. I ran the following code:
print(hex(id(df)))
print(hex(id(df.sample(frac=1))))
print(hex(id(df.sample(frac=1).reset_index(drop=True))))
and my results were:
0x1f8a784d400
0x1f8b9d65e10
0x1f8b9d65b70
which means the method is not returning the same object, as was suggested in the last comment. So this method does indeed make a shuffled copy.
Here is another way:
df['rnd'] = np.random.rand(len(df))
df = df.sort_values(by='rnd', inplace=True).drop('rnd', axis=1)
What is also useful, if you use it for Machine_learning and want to separate always the same data, you could use:
df.sample(n=len(df), random_state=42)
This makes sure, that you keep your random choice always replicable
Following could be one of ways:
dataframe = dataframe.sample(frac=1, random_state=42).reset_index(drop=True)
where
frac=1 means all rows of a data frame
random_state=42 means keeping the same order in each execution
reset_index(drop=True) means reinitialize index for randomized dataframe
Shuffle the DataFrame using sample() by passing the frac
parameter.
Save the shuffled DataFrame to a new variable.
new_variable = DataFrame.sample(frac=1)
I propose this:
for x in df.columns:
np.random.seed(42);
np.random.shuffle(df[x].values)
With my test with a column of arbitrary length strings (with dtype: object
), it was 30x faster than @haku’s answer, presumably because it avoids creating a copy which may be expensive.
My variant was about 3x faster than the accepted @Kris’es answer which also does not seem to avoid a copy (based on RES
column in Linux top
).
Without numpy/sklean 🙂 and in case you want to shuffle all values, but keep rows & columns names in place.
df_c = df.copy()
df_c.iloc[:,:] = df_c.sample(frac=1,random_state=123,ignore_index=True)
I have the following DataFrame:
Col1 Col2 Col3 Type
0 1 2 3 1
1 4 5 6 1
...
20 7 8 9 2
21 10 11 12 2
...
45 13 14 15 3
46 16 17 18 3
...
The DataFrame is read from a CSV file. All rows which have Type
1 are on top, followed by the rows with Type
2, followed by the rows with Type
3, etc.
I would like to shuffle the order of the DataFrame’s rows so that all Type
‘s are mixed. A possible result could be:
Col1 Col2 Col3 Type
0 7 8 9 2
1 13 14 15 3
...
20 1 2 3 1
21 10 11 12 2
...
45 4 5 6 1
46 16 17 18 3
...
How can I achieve this?
You can shuffle the rows of a data frame by indexing with a shuffled index. For this, you can eg use np.random.permutation
(but np.random.choice
is also a possibility):
In [12]: df = pd.read_csv(StringIO(s), sep="s+")
In [13]: df
Out[13]:
Col1 Col2 Col3 Type
0 1 2 3 1
1 4 5 6 1
20 7 8 9 2
21 10 11 12 2
45 13 14 15 3
46 16 17 18 3
In [14]: df.iloc[np.random.permutation(len(df))]
Out[14]:
Col1 Col2 Col3 Type
46 16 17 18 3
45 13 14 15 3
20 7 8 9 2
0 1 2 3 1
1 4 5 6 1
21 10 11 12 2
If you want to keep the index numbered from 1, 2, .., n as in your example, you can simply reset the index: df_shuffled.reset_index(drop=True)
The idiomatic way to do this with Pandas is to use the .sample
method of your data frame to sample all rows without replacement:
df.sample(frac=1)
The frac
keyword argument specifies the fraction of rows to return in the random sample, so frac=1
means to return all rows (in random order).
Note:
If you wish to shuffle your dataframe in-place and reset the index, you could do e.g.
df = df.sample(frac=1).reset_index(drop=True)
Here, specifying drop=True
prevents .reset_index
from creating a column containing the old index entries.
Follow-up note: Although it may not look like the above operation is in-place, python/pandas is smart enough not to do another malloc for the shuffled object. That is, even though the reference object has changed (by which I mean id(df_old)
is not the same as id(df_new)
), the underlying C object is still the same. To show that this is indeed the case, you could run a simple memory profiler:
$ python3 -m memory_profiler .test.py
Filename: .test.py
Line # Mem usage Increment Line Contents
================================================
5 68.5 MiB 68.5 MiB @profile
6 def shuffle():
7 847.8 MiB 779.3 MiB df = pd.DataFrame(np.random.randn(100, 1000000))
8 847.9 MiB 0.1 MiB df = df.sample(frac=1).reset_index(drop=True)
You can simply use sklearn
for this
from sklearn.utils import shuffle
df = shuffle(df)
TL;DR: np.random.shuffle(ndarray)
can do the job.
So, in your case
np.random.shuffle(DataFrame.values)
DataFrame
, under the hood, uses NumPy ndarray as a data holder. (You can check from DataFrame source code)
So if you use np.random.shuffle()
, it would shuffle the array along the first axis of a multi-dimensional array. But the index of the DataFrame
remains unshuffled.
Though, there are some points to consider.
- function returns none. In case you want to keep a copy of the original object, you have to do so before you pass to the function.
sklearn.utils.shuffle()
, as user tj89 suggested, can designaterandom_state
along with another option to control output. You may want that for dev purposes.sklearn.utils.shuffle()
is faster. But WILL SHUFFLE the axis info(index, column) of theDataFrame
along with thendarray
it contains.
Benchmark result
between sklearn.utils.shuffle()
and np.random.shuffle()
.
ndarray
nd = sklearn.utils.shuffle(nd)
0.10793248389381915 sec. 8x faster
np.random.shuffle(nd)
0.8897626010002568 sec
DataFrame
df = sklearn.utils.shuffle(df)
0.3183923360193148 sec. 3x faster
np.random.shuffle(df.values)
0.9357550159329548 sec
Conclusion: If it is okay to axis info(index, column) to be shuffled along with ndarray, use
sklearn.utils.shuffle()
. Otherwise, usenp.random.shuffle()
used code
import timeit
setup = '''
import numpy as np
import pandas as pd
import sklearn
nd = np.random.random((1000, 100))
df = pd.DataFrame(nd)
'''
timeit.timeit('nd = sklearn.utils.shuffle(nd)', setup=setup, number=1000)
timeit.timeit('np.random.shuffle(nd)', setup=setup, number=1000)
timeit.timeit('df = sklearn.utils.shuffle(df)', setup=setup, number=1000)
timeit.timeit('np.random.shuffle(df.values)', setup=setup, number=1000)
shuffle the pandas data frame by taking a sample array in this case index and randomize its order then set the array as an index of data frame. Now sort the data frame according to index. Here goes your shuffled dataframe
import random
df = pd.DataFrame({"a":[1,2,3,4],"b":[5,6,7,8]})
index = [i for i in range(df.shape[0])]
random.shuffle(index)
df.set_index([index]).sort_index()
output
a b
0 2 6
1 1 5
2 3 7
3 4 8
Insert you data frame in the place of mine in above code .
Here is another way to do this:
df_shuffled = df.reindex(np.random.permutation(df.index))
(I don’t have enough reputation to comment this on the top post, so I hope someone else can do that for me.) There was a concern raised that the first method:
df.sample(frac=1)
It makes a deep copy or just changed the dataframe. I ran the following code:
print(hex(id(df)))
print(hex(id(df.sample(frac=1))))
print(hex(id(df.sample(frac=1).reset_index(drop=True))))
and my results were:
0x1f8a784d400
0x1f8b9d65e10
0x1f8b9d65b70
which means the method is not returning the same object, as was suggested in the last comment. So this method does indeed make a shuffled copy.
Here is another way:
df['rnd'] = np.random.rand(len(df))
df = df.sort_values(by='rnd', inplace=True).drop('rnd', axis=1)
What is also useful, if you use it for Machine_learning and want to separate always the same data, you could use:
df.sample(n=len(df), random_state=42)
This makes sure, that you keep your random choice always replicable
Following could be one of ways:
dataframe = dataframe.sample(frac=1, random_state=42).reset_index(drop=True)
where
frac=1 means all rows of a data frame
random_state=42 means keeping the same order in each execution
reset_index(drop=True) means reinitialize index for randomized dataframe
Shuffle the DataFrame using sample() by passing the frac
parameter.
Save the shuffled DataFrame to a new variable.
new_variable = DataFrame.sample(frac=1)
I propose this:
for x in df.columns:
np.random.seed(42);
np.random.shuffle(df[x].values)
With my test with a column of arbitrary length strings (with dtype: object
), it was 30x faster than @haku’s answer, presumably because it avoids creating a copy which may be expensive.
My variant was about 3x faster than the accepted @Kris’es answer which also does not seem to avoid a copy (based on RES
column in Linux top
).
Without numpy/sklean 🙂 and in case you want to shuffle all values, but keep rows & columns names in place.
df_c = df.copy()
df_c.iloc[:,:] = df_c.sample(frac=1,random_state=123,ignore_index=True)