How to force a new line when appending to a csv using python pandas .to_csv

Question

When appending to csv, my first line is starting on the existing last line rather than a new line.

I keep searching SO, but I am just finding the basic use of opening a csv in append mode or using append mode when writing to csv. I could not make sense of the accepted answer here (to_csv append mode is not appending to next new line) since it appears to require the existing file to be open before writing the (“/n”) with f.write("/n"). This answer (How to add pandas data to an existing csv file?) is most relevant, but I am hoping to write multiple data frames in a function, so I do not want to keep opening them. My plan is to use a function like:

import os
def mysave(df,dfpath):
    # if file does not exist write header 
    if not os.path.isfile(dfpath):
        df.to_csv(dfpath, index = False)
    else: # else it exists so append without writing the header
        df.to_csv(dfpath, mode = 'a', index = False, header = False)

mysave(mydf, 'foo.csv')

I’ve created a very simple example, with foo.csv with the structure:

a   b   c   d           
5   1   ah  doo         
6   2   bah poo         
7   2   dah coo

When I use my function or this simple code:

import pandas as pd
df = pd.read_csv('foo.csv', index_col=False)
mydf = df
mydf.to_csv('foo.csv', mode='a', index = False, header = False)

This is what foo.csv ends up as:

a   b   c   d           
5   1   ah  doo         
6   2   bah poo         
7   2   dah coo5    1   ah  doo
6   2   bah poo         
7   2   dah coo

When I attempt to add a carriage return character as the header, like mydf.to_csv('foo.csv', mode='a', index = False, header = ("/n"))
pandas (rightly) ignores my erroneous header comment and goes with the default of header = True.

a   b   c   d           
5   1   ah  doo         
6   2   bah poo         
7   2   dah cooa    b   c   d
6   2   bah poo         
7   2   dah coo

Asked By: jessi

||

Source

Answer 1

I am assuming that you are going to appending one below other of two dataframe into single dataframe.

use below mentioned command to make it as single command

ans = pd.concat([df, df])

then you can make output into .csv file

Answered By: sriramkumar

Answer 2

If your dataframe gets huge and you want to avoid concatenation you could go with

import csv
with open('foo.csv','ab') as out:
   writer=csv.writer(out)
   writer.writerow(())

in a function or just as a snippet in your code. If you’re not on Windows maybe you could avoid adding ‘b’ in open and open the file with just ‘a’ (append)

Answered By: themistoklik

Answer 3

I had a similar problem and after a good bit of searching, I didn’t find any simple/elegant solution. The minimal fix that worked for me is:

import pandas as pd

with open('foo.csv', 'a') as f:
    f.write('n')    
mydf.to_csv('foo.csv', index = False, header = False, mode='a')

Answered By: mathamateur

How to force a new line when appending to a csv using python pandas .to_csv

Question:

Answers: