python read csv files with same basename and save as different dataframes

Question

I have 20 csv files with the same basename and a number from 100 to 2000 with an increment of 100 between files, such that samp_100.csv, samp_200.csv, samp_300.csv, …, samp_1900.csv, samp_2000.csv.

I am trying to read these files into python. I am trying the following.

T = np.arange(100,2100,100)
for i in T: 
    df = pd.read_csv("samp_{i}.csv".format(i=i))

Although I do not get an error, the files aren’t read in the correct order from 100 to 2000. When I use df.head, I do not see the first lines of the file samp_100.csv. Also the files are concatenated into a single file called df. Is there an equivalent way to achieve this but instead have 20 separate dataframes with the names df_100, df_200, …, df_1900, df_2000?

Asked By: user19619903

||

Source

Answer 1

You need pandas.concat.

Try this :

import numpy as np
import pandas as pd

T = np.arange(100,2100,100)

list_of_df = []
for i in T:
    temp_df = pd.read_csv(f"samp_{i}.csv")
    list_of_df.append(temp_df)
    
df = pd.concat(list_of_df, axis=0, ignore_index=True)

If you need to add a column with the name of the .csv, include the line below after calling pandas.read_csv inside the loop.

temp_df.insert(0, "filename", f"samp_{i}")

Answered By: abokey

python read csv files with same basename and save as different dataframes

Question:

Answers: