Good alternative to Pandas .append() method, now that it is being deprecated?
Question:
I use the following method a lot to append a single row to a dataframe. One thing I really like about it is that it allows you to append a simple dict object. For example:
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:
# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])
So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. 🙁 Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?
Answers:
Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List’s "append" method are very efficient and won’t be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design – that is why they deprecated the method
I also like the append method.
But you can do it in one line with a list of dicts
df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])
or using loc and tuples for values on DataFrames with incremenal ascending indexes
df.loc[len(df), ['a','b']] = 1, 2
or maybe
df.loc[len(df), df.columns] = 3, 4
If you want to use concat instead:
append
outputxlsx = outputxlsx.append(df, ignore_index=True)
concat
outputxlsx = pd.concat([outputxlsx, df])
i also have the problem when using DataFrame.append in my program before, but it has been fixed now. Hopefully this snippet can help so.
import pandas as pd
df1=pd.DataFrame(dict_1)
def addData(param1,param2,param3):
dict_2={"list1":var1, "list2":var2, "list3":var3}
df2=pdDataFrame(dict_2, index={len(dict_2)+1})
dfc=pd.concat([df1, df2])
return dfc
I was facing a similar issue. The other solutions weren’t really working for me. I’m leaving this answer here as an additional possibility to deal with the issue since this is the first google result for certain searches and I myself ended here at least for the second time.
In my case the data is not a dict but just a list of values for a known set of parameters.
I want to add the parameter values to a dataframe as rows because this way I can access a series of all the values for one parameter via df[parameter].
I start with an empty DataFrame:
parameters = ['a', 'b', 'c', 'd', 'e', 'f']
df = pd.DataFrame(columns=parameters)
df:
a b c d e f
With append I could add rows very convenient like so:
new_row = pd.Series([1,2,3,4,5,6], index=parameters, name='row1')
df.append(new_row)
df:
a b c d e f
row1 1 2 3 4 5 6
With pd.concat I found this to deliver the same result in very similar way:
new_row = pd.DataFrame([1,2,3,4,5,6], columns=['row1'], index=parameters).T
df = pd.concat((df, new_row))
The key was to create a (1,n) dataframe from the 1d data and then transpose it to match the other dataframe.
# Deprecated issue has been resolved
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
print("df columns:", df)
# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
print("df column Values :", df)
# Create the new row as its own dataframe
df_new_row = pd.DataFrame.from_records({ 'a': [3], 'b': [4] })
df = pd.concat([df, df_new_row])
print("pd concat with two df's :", df)
For those, like me, who want a descriptive function rather than lots of one-liners, here is an option based on @Rafael Gaitan above.
def appendDictToDF(df,dictToAppend):
df = pd.concat([df, pd.DataFrame.from_records([dictToAppend])])
return df
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
# Appending a row
df= appendDictToDF(df,{ 'a': 1, 'b': 2 })
Python at() method enables us to update the value of one row at a time with respect to a column.
dataframe.at[index,'column-name']='new value'
I use the following method a lot to append a single row to a dataframe. One thing I really like about it is that it allows you to append a simple dict object. For example:
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:
# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])
So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. 🙁 Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?
Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List’s "append" method are very efficient and won’t be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design – that is why they deprecated the method
I also like the append method.
But you can do it in one line with a list of dicts
df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])
or using loc and tuples for values on DataFrames with incremenal ascending indexes
df.loc[len(df), ['a','b']] = 1, 2
or maybe
df.loc[len(df), df.columns] = 3, 4
If you want to use concat instead:
append
outputxlsx = outputxlsx.append(df, ignore_index=True)
concat
outputxlsx = pd.concat([outputxlsx, df])
i also have the problem when using DataFrame.append in my program before, but it has been fixed now. Hopefully this snippet can help so.
import pandas as pd
df1=pd.DataFrame(dict_1)
def addData(param1,param2,param3):
dict_2={"list1":var1, "list2":var2, "list3":var3}
df2=pdDataFrame(dict_2, index={len(dict_2)+1})
dfc=pd.concat([df1, df2])
return dfc
I was facing a similar issue. The other solutions weren’t really working for me. I’m leaving this answer here as an additional possibility to deal with the issue since this is the first google result for certain searches and I myself ended here at least for the second time.
In my case the data is not a dict but just a list of values for a known set of parameters.
I want to add the parameter values to a dataframe as rows because this way I can access a series of all the values for one parameter via df[parameter].
I start with an empty DataFrame:
parameters = ['a', 'b', 'c', 'd', 'e', 'f']
df = pd.DataFrame(columns=parameters)
df:
a b c d e f
With append I could add rows very convenient like so:
new_row = pd.Series([1,2,3,4,5,6], index=parameters, name='row1')
df.append(new_row)
df:
a b c d e f
row1 1 2 3 4 5 6
With pd.concat I found this to deliver the same result in very similar way:
new_row = pd.DataFrame([1,2,3,4,5,6], columns=['row1'], index=parameters).T
df = pd.concat((df, new_row))
The key was to create a (1,n) dataframe from the 1d data and then transpose it to match the other dataframe.
# Deprecated issue has been resolved
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
print("df columns:", df)
# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
print("df column Values :", df)
# Create the new row as its own dataframe
df_new_row = pd.DataFrame.from_records({ 'a': [3], 'b': [4] })
df = pd.concat([df, df_new_row])
print("pd concat with two df's :", df)
For those, like me, who want a descriptive function rather than lots of one-liners, here is an option based on @Rafael Gaitan above.
def appendDictToDF(df,dictToAppend):
df = pd.concat([df, pd.DataFrame.from_records([dictToAppend])])
return df
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
# Appending a row
df= appendDictToDF(df,{ 'a': 1, 'b': 2 })
Python at() method enables us to update the value of one row at a time with respect to a column.
dataframe.at[index,'column-name']='new value'