why should I make a copy of a data frame in pandas
Question:
When selecting a sub dataframe from a parent dataframe, I noticed that some programmers make a copy of the data frame using the .copy() method. For example,
X = my_dataframe[features_list].copy()
…instead of just
X = my_dataframe[features_list]
Why are they making a copy of the data frame? What will happen if I don’t make a copy?
Answers:
This expands on Paul’s answer. In Pandas, indexing a DataFrame returns a reference to the initial DataFrame. Thus, changing the subset will change the initial DataFrame. Thus, you’d want to use the copy if you want to make sure the initial DataFrame shouldn’t change. Consider the following code:
df = DataFrame({'x': [1,2]})
df_sub = df[0:1]
df_sub.x = -1
print(df)
You’ll get:
x
0 -1
1 2
In contrast, the following leaves df unchanged:
df_sub_copy = df[0:1].copy()
df_sub_copy.x = -1
This answer has been deprecated in newer versions of pandas. See docs
Because if you don’t make a copy then the indices can still be manipulated elsewhere even if you assign the dataFrame to a different name.
For example:
df2 = df
func1(df2)
func2(df)
func1 can modify df by modifying df2, so to avoid that:
df2 = df.copy()
func1(df2)
func2(df)
It’s necessary to mention that returning copy or view depends on kind of indexing.
The pandas documentation says:
Returning a view versus a copy
The rules about when a view on the data is returned are entirely
dependent on NumPy. Whenever an array of labels or a boolean vector
are involved in the indexing operation, the result will be a copy.
With single label / scalar indexing and slicing, e.g. df.ix[3:6] or
df.ix[:, ‘A’], a view will be returned.
In general it is safer to work on copies than on original data frames, except when you know that you won’t be needing the original anymore and want to proceed with the manipulated version. Normally, you would still have some use for the original data frame to compare with the manipulated version, etc. Therefore, most people work on copies and merge at the end.
The primary purpose is to avoid chained indexing and eliminate the SettingWithCopyWarning.
Here chained indexing is something like dfc['A'][0] = 111
The document said chained indexing should be avoided in Returning a view versus a copy. Here is a slightly modified example from that document:
In [1]: import pandas as pd
In [2]: dfc = pd.DataFrame({'A':['aaa','bbb','ccc'],'B':[1,2,3]})
In [3]: dfc
Out[3]:
A B
0 aaa 1
1 bbb 2
2 ccc 3
In [4]: aColumn = dfc['A']
In [5]: aColumn[0] = 111
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
In [6]: dfc
Out[6]:
A B
0 111 1
1 bbb 2
2 ccc 3
Here the aColumn is a view and not a copy from the original DataFrame, so modifying aColumn will cause the original dfc be modified too. Next, if we index the row first:
In [7]: zero_row = dfc.loc[0]
In [8]: zero_row['A'] = 222
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
In [9]: dfc
Out[9]:
A B
0 111 1
1 bbb 2
2 ccc 3
This time zero_row is a copy, so the original dfc is not modified.
From these two examples above, we see it’s ambiguous whether or not you want to change the original DataFrame. This is especially dangerous if you write something like the following:
In [10]: dfc.loc[0]['A'] = 333
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
In [11]: dfc
Out[11]:
A B
0 111 1
1 bbb 2
2 ccc 3
This time it didn’t work at all. Here we wanted to change dfc, but we actually modified an intermediate value dfc.loc[0] that is a copy and is discarded immediately. It’s very hard to predict whether the intermediate value like dfc.loc[0] or dfc['A'] is a view or a copy, so it’s not guaranteed whether or not original DataFrame will be updated. That’s why chained indexing should be avoided, and pandas generates the SettingWithCopyWarning for this kind of chained indexing update.
Now is the use of .copy(). To eliminate the warning, make a copy to express your intention explicitly:
In [12]: zero_row_copy = dfc.loc[0].copy()
In [13]: zero_row_copy['A'] = 444 # This time no warning
Since you are modifying a copy, you know the original dfc will never change and you are not expecting it to change. Your expectation matches the behavior, then the SettingWithCopyWarning disappears.
Note, If you do want to modify the original DataFrame, the document suggests you use loc:
In [14]: dfc.loc[0,'A'] = 555
In [15]: dfc
Out[15]:
A B
0 555 1
1 bbb 2
2 ccc 3
Assumed you have data frame as below
df1
A B C D
4 -1.0 -1.0 -1.0 -1.0
5 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
When you would like create another df2 which is identical to df1, without copy
df2=df1
df2
A B C D
4 -1.0 -1.0 -1.0 -1.0
5 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
And would like modify the df2 value only as below
df2.iloc[0,0]='changed'
df2
A B C D
4 changed -1.0 -1.0 -1.0
5 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
At the same time the df1 is changed as well
df1
A B C D
4 changed -1.0 -1.0 -1.0
5 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
Since two df as same object, we can check it by using the id
id(df1)
140367679979600
id(df2)
140367679979600
So they as same object and one change another one will pass the same value as well.
If we add the copy, and now df1 and df2 are considered as different object, if we do the same change to one of them the other will not change.
df2=df1.copy()
id(df1)
140367679979600
id(df2)
140367674641232
df1.iloc[0,0]='changedback'
df2
A B C D
4 changed -1.0 -1.0 -1.0
5 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
Good to mention, when you subset the original dataframe, it is safe to add the copy as well in order to avoid the SettingWithCopyWarning
Pandas Deep copy leaves the initial DataFrame unchanged.
This feature is particularly useful when you want to normalize a DataFrame and want to keep the initial df unchanged.
For instance:
df = pd.DataFrame(np.arange(20).reshape(2,10))
then you normalize the data:
# Using Sklearn MinMaxSacaler method
scaler = preprocessing.MinMaxScaler()
and you make a new df based on the first one and want the first one unchanged,
you have to use .copy() method
new_df = pd.DataFrame(df).copy() # Deep Copy
for i in range(10):
pd_features[i] = scaler.fit_transform(unnormal_pd_features[i].values.reshape(-1,1))
or else your original df will change too.
I was so careless using copy() until I use that line of code below
without using copy(), the changes in df_genel3 effects df_genel
df_genel3 = df_genel
df_genel3.loc[(df_genel3['Hareket']=='İmha') , 'Hareket_Tutar'] = tutar
copy() solved the problem
df_genel3 = df_genel.copy()
df_genel3.loc[(df_genel3['Hareket']=='İmha') , 'Hareket_Tutar'] = tutar
When selecting a sub dataframe from a parent dataframe, I noticed that some programmers make a copy of the data frame using the .copy() method. For example,
X = my_dataframe[features_list].copy()
…instead of just
X = my_dataframe[features_list]
Why are they making a copy of the data frame? What will happen if I don’t make a copy?
This expands on Paul’s answer. In Pandas, indexing a DataFrame returns a reference to the initial DataFrame. Thus, changing the subset will change the initial DataFrame. Thus, you’d want to use the copy if you want to make sure the initial DataFrame shouldn’t change. Consider the following code:
df = DataFrame({'x': [1,2]})
df_sub = df[0:1]
df_sub.x = -1
print(df)
You’ll get:
x
0 -1
1 2
In contrast, the following leaves df unchanged:
df_sub_copy = df[0:1].copy()
df_sub_copy.x = -1
This answer has been deprecated in newer versions of pandas. See docs
Because if you don’t make a copy then the indices can still be manipulated elsewhere even if you assign the dataFrame to a different name.
For example:
df2 = df
func1(df2)
func2(df)
func1 can modify df by modifying df2, so to avoid that:
df2 = df.copy()
func1(df2)
func2(df)
It’s necessary to mention that returning copy or view depends on kind of indexing.
The pandas documentation says:
Returning a view versus a copy
The rules about when a view on the data is returned are entirely
dependent on NumPy. Whenever an array of labels or a boolean vector
are involved in the indexing operation, the result will be a copy.
With single label / scalar indexing and slicing, e.g. df.ix[3:6] or
df.ix[:, ‘A’], a view will be returned.
In general it is safer to work on copies than on original data frames, except when you know that you won’t be needing the original anymore and want to proceed with the manipulated version. Normally, you would still have some use for the original data frame to compare with the manipulated version, etc. Therefore, most people work on copies and merge at the end.
The primary purpose is to avoid chained indexing and eliminate the SettingWithCopyWarning.
Here chained indexing is something like dfc['A'][0] = 111
The document said chained indexing should be avoided in Returning a view versus a copy. Here is a slightly modified example from that document:
In [1]: import pandas as pd
In [2]: dfc = pd.DataFrame({'A':['aaa','bbb','ccc'],'B':[1,2,3]})
In [3]: dfc
Out[3]:
A B
0 aaa 1
1 bbb 2
2 ccc 3
In [4]: aColumn = dfc['A']
In [5]: aColumn[0] = 111
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
In [6]: dfc
Out[6]:
A B
0 111 1
1 bbb 2
2 ccc 3
Here the aColumn is a view and not a copy from the original DataFrame, so modifying aColumn will cause the original dfc be modified too. Next, if we index the row first:
In [7]: zero_row = dfc.loc[0]
In [8]: zero_row['A'] = 222
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
In [9]: dfc
Out[9]:
A B
0 111 1
1 bbb 2
2 ccc 3
This time zero_row is a copy, so the original dfc is not modified.
From these two examples above, we see it’s ambiguous whether or not you want to change the original DataFrame. This is especially dangerous if you write something like the following:
In [10]: dfc.loc[0]['A'] = 333
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
In [11]: dfc
Out[11]:
A B
0 111 1
1 bbb 2
2 ccc 3
This time it didn’t work at all. Here we wanted to change dfc, but we actually modified an intermediate value dfc.loc[0] that is a copy and is discarded immediately. It’s very hard to predict whether the intermediate value like dfc.loc[0] or dfc['A'] is a view or a copy, so it’s not guaranteed whether or not original DataFrame will be updated. That’s why chained indexing should be avoided, and pandas generates the SettingWithCopyWarning for this kind of chained indexing update.
Now is the use of .copy(). To eliminate the warning, make a copy to express your intention explicitly:
In [12]: zero_row_copy = dfc.loc[0].copy()
In [13]: zero_row_copy['A'] = 444 # This time no warning
Since you are modifying a copy, you know the original dfc will never change and you are not expecting it to change. Your expectation matches the behavior, then the SettingWithCopyWarning disappears.
Note, If you do want to modify the original DataFrame, the document suggests you use loc:
In [14]: dfc.loc[0,'A'] = 555
In [15]: dfc
Out[15]:
A B
0 555 1
1 bbb 2
2 ccc 3
Assumed you have data frame as below
df1
A B C D
4 -1.0 -1.0 -1.0 -1.0
5 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
When you would like create another df2 which is identical to df1, without copy
df2=df1
df2
A B C D
4 -1.0 -1.0 -1.0 -1.0
5 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
6 -1.0 -1.0 -1.0 -1.0
And would like modify the df2 value only as below
df2.iloc[0,0]='changed'
df2
A B C D
4 changed -1.0 -1.0 -1.0
5 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
At the same time the df1 is changed as well
df1
A B C D
4 changed -1.0 -1.0 -1.0
5 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
Since two df as same object, we can check it by using the id
id(df1)
140367679979600
id(df2)
140367679979600
So they as same object and one change another one will pass the same value as well.
If we add the copy, and now df1 and df2 are considered as different object, if we do the same change to one of them the other will not change.
df2=df1.copy()
id(df1)
140367679979600
id(df2)
140367674641232
df1.iloc[0,0]='changedback'
df2
A B C D
4 changed -1.0 -1.0 -1.0
5 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
6 -1 -1.0 -1.0 -1.0
Good to mention, when you subset the original dataframe, it is safe to add the copy as well in order to avoid the SettingWithCopyWarning
Pandas Deep copy leaves the initial DataFrame unchanged.
This feature is particularly useful when you want to normalize a DataFrame and want to keep the initial df unchanged.
For instance:
df = pd.DataFrame(np.arange(20).reshape(2,10))
then you normalize the data:
# Using Sklearn MinMaxSacaler method
scaler = preprocessing.MinMaxScaler()
and you make a new df based on the first one and want the first one unchanged,
you have to use .copy() method
new_df = pd.DataFrame(df).copy() # Deep Copy
for i in range(10):
pd_features[i] = scaler.fit_transform(unnormal_pd_features[i].values.reshape(-1,1))
or else your original df will change too.
I was so careless using copy() until I use that line of code below
without using copy(), the changes in df_genel3 effects df_genel
df_genel3 = df_genel
df_genel3.loc[(df_genel3['Hareket']=='İmha') , 'Hareket_Tutar'] = tutar
copy() solved the problem
df_genel3 = df_genel.copy()
df_genel3.loc[(df_genel3['Hareket']=='İmha') , 'Hareket_Tutar'] = tutar