Save dataframe to file so that it can be reimported

Question:

I’m quite new to Python and I’m wondering what would be the best way to save a dataframe in a way so that it can be easily reimported. I’m running a code that checks for newly added files and imports them. In case it is stopped for any reason, I would like to reimport the last data that was saved before it stopped.

So, the data I’m interested in is in avg_data. To save it to txt, I use:

df_file_name = folder_path + '/' + file_name + '_lastData.txt'
        
        with open(df_file_name, "w") as new_file:
             avg_data.to_string(new_file, index=False)

But with this I struggle to properly reimport.. I use:

previous_data_str = folder_path + '/' + file_name + '_lastData.txt'
if os.path.isfile(previous_data_str) :
    old_data = pd.read_csv(previous_data_str)

to reimport, but this doesn’t work… I don’t get an error message but the data is now all in one column… using sep="t" leads to the same, using sep=" " leads to too many empty columns… I uploaded a sample .txt file here

Not sure what I’m doing wrong here… and if it is even the best to save it in this way. I would greatly appreciate some tips on how to handle this best, thanks!

Asked By: Data123

||

Answers:

Simply save it to csv with pandas.DataFrame.to_csv and then you can easily import it with pandas.read_csv.

Answered By: yeti
Categories: questions Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.