Panda's Write CSV – Append vs. Write

Question

I would like to use pd.write_csv to write “filename” (with headers) if “filename” doesn’t exist, otherwise to append to “filename” if it exists. If I simply use command:

     df.to_csv('filename.csv',mode = 'a',header ='column_names')

The write or append succeeds, but it seems like the header is written every time an append takes place.

How can I only add the header if the file doesn’t exist, and append without header if the file does exist?

Asked By: GPB

||

Source

Answer 1

Not sure there is a way in pandas but checking if the file exists would be a simple approach:

import os
# if file does not exist write header 
if not os.path.isfile('filename.csv'):
   df.to_csv('filename.csv', header='column_names')
else: # else it exists so append without writing the header
   df.to_csv('filename.csv', mode='a', header=False)

Answered By: Padraic Cunningham

Answer 2

with open(filename, 'a') as f:
    df.to_csv(f, mode='a', header=f.tell()==0)

it will add header when writes to the file first time

Answered By: user3657041

Answer 3

In Pandas dataframe "to_csv" function, use header=False if csv file exists & append to existing file.

import os

hdr = False  if os.path.isfile('filename.csv') else True
df.to_csv('filename.csv', mode='a', header=hdr)

Answered By: VK Singh

Answer 4

The above solutions are great, but I have a moral obligation to include the pathlib solution here:

from pathlib import Path

file_path = Path(filename)
if file_path.exists():
   df.to_csv(file_path, header=False, mode='a')
else:
   df.to_csv(file_path, header=True, mode='w')

Alternatively (depending on your inlining preferences):

file_exists = file_path.exists()
df.to_csv(file_path, header=not file_exists, mode='a' if file_exists else 'w')

Answered By: DV82XL

Answer 5

Apart from file exist check, you can also check for non zero file size. Since it will make sense to add header if file exists but file size is zero i.e file without content. I find it helpful in some exceptional cases

import os.path
header_flag = False if (os.path.exists(fpath) and (os.path.getsize(fpath) > 0)) else True
df.to_csv(fpath, mode='a', index=False, header=header_flag)

Answered By: Alok Nayak

Answer 6

In case if you have dict() and want to write and append into CSV file :

import pandas as pd

file_name = 'data.csv'

my_dict = {"column_1":"Apple","column_2":"Mango"}

with open(file_name, 'a') as f:
  
  df = pd.DataFrame(my_dict)
  df.to_csv(f, mode='a', header=f.tell()==0)

Answered By: Vinay Chaudhari

Panda's Write CSV – Append vs. Write

Question:

Answers: