Replace rows in a Pandas df with rows from another df
Question:
I have 2 Pandas dfs, A and B. Both have 10 columns and the index ‘ID’. Where the IDs of A and B match, I want to replace the rows of B with the rows of A. I have tried to use pd.update, but no success yet. Any help appreciated.
Answers:
below code should do the trick
s1 = pd.Series([5, 1, 'a'])
s2 = pd.Series([6, 2, 'b'])
s3 = pd.Series([7, 3, 'd'])
s4 = pd.Series([8, 4, 'e'])
s5 = pd.Series([9, 5, 'f'])
df1 = pd.DataFrame([list(s1), list(s2),list(s3),list(s4),list(s5)], columns = ["A", "B", "C"])
s1 = pd.Series([5, 6, 'p'])
s2 = pd.Series([6, 7, 'q'])
s3 = pd.Series([7, 8, 'r'])
s4 = pd.Series([8, 9, 's'])
s5 = pd.Series([9, 10, 't'])
df2 = pd.DataFrame([list(s1), list(s2),list(s3),list(s4),list(s5)], columns = ["A", "B", "C"])
df1.loc[df1.A.isin(df2.A), ['B', 'C']] = df2[['B', 'C']]
print df1
output
A B C
0 5 6 p
1 6 7 q
2 7 8 r
3 8 9 s
4 9 10 t
Edit from comments:
To replace the whole row instead of only some columns:
cols = list(df1.columns)
df1.loc[df1.A.isin(df2.A), cols] = df2[cols]
You can empty your target cells in A (by setting them to NaN) and use the combine_first() method to fill those with B‘s values. Although it may sound counter-intuitive, this approach gives you the flexibility to both target rows and specific columns in 2 lines of code. Hope that helps.
An example replacing the full row’s that have an index match:
# set-up
cols = ['c1','c2','c3']
A = pd.DataFrame(np.arange(9).reshape((3,3)), columns=cols)
B = pd.DataFrame(np.arange(10,16).reshape((2,3)), columns=cols)
#solution
A.loc[B.index] = np.nan
A = A.combine_first(B)
An example of only replacing certain target columns for row’s that have an index match:
A.loc[B.index, ['c2','c3']] = np.nan
A = A.combine_first(B)
Above code works based on the index value.
If we have different row counts for both data frame. It will not work.
For that, we need to set a specific column to index
I have 2 Pandas dfs, A and B. Both have 10 columns and the index ‘ID’. Where the IDs of A and B match, I want to replace the rows of B with the rows of A. I have tried to use pd.update, but no success yet. Any help appreciated.
below code should do the trick
s1 = pd.Series([5, 1, 'a'])
s2 = pd.Series([6, 2, 'b'])
s3 = pd.Series([7, 3, 'd'])
s4 = pd.Series([8, 4, 'e'])
s5 = pd.Series([9, 5, 'f'])
df1 = pd.DataFrame([list(s1), list(s2),list(s3),list(s4),list(s5)], columns = ["A", "B", "C"])
s1 = pd.Series([5, 6, 'p'])
s2 = pd.Series([6, 7, 'q'])
s3 = pd.Series([7, 8, 'r'])
s4 = pd.Series([8, 9, 's'])
s5 = pd.Series([9, 10, 't'])
df2 = pd.DataFrame([list(s1), list(s2),list(s3),list(s4),list(s5)], columns = ["A", "B", "C"])
df1.loc[df1.A.isin(df2.A), ['B', 'C']] = df2[['B', 'C']]
print df1
output
A B C
0 5 6 p
1 6 7 q
2 7 8 r
3 8 9 s
4 9 10 t
Edit from comments:
To replace the whole row instead of only some columns:
cols = list(df1.columns)
df1.loc[df1.A.isin(df2.A), cols] = df2[cols]
You can empty your target cells in A (by setting them to NaN) and use the combine_first() method to fill those with B‘s values. Although it may sound counter-intuitive, this approach gives you the flexibility to both target rows and specific columns in 2 lines of code. Hope that helps.
An example replacing the full row’s that have an index match:
# set-up
cols = ['c1','c2','c3']
A = pd.DataFrame(np.arange(9).reshape((3,3)), columns=cols)
B = pd.DataFrame(np.arange(10,16).reshape((2,3)), columns=cols)
#solution
A.loc[B.index] = np.nan
A = A.combine_first(B)
An example of only replacing certain target columns for row’s that have an index match:
A.loc[B.index, ['c2','c3']] = np.nan
A = A.combine_first(B)
Above code works based on the index value.
If we have different row counts for both data frame. It will not work.
For that, we need to set a specific column to index