How to append data to excel into single excel sheet using for loop

Question

I am trying to read log files from different folder and store in excel sheet

import os
import pandas as pd
import bs4

path = "D:\logfolder"
filelist = []

for root, dirs, files in os.walk(path):
    for file in files:
        if file.endswith("History.txt"):
            filelist.append(os.path.join(root, file))

for name in filelist:             
        print(name + "n")   # D:logfolderlogfolder_1History.txt
                             # D:logfolderlogfolder_2History.txt
                             # D:logfolderlogfolder_3History.txt
                             # D:logfolderlogfolder_4History.txt


for name in filelist:
    with open(name,"r") as f:
        soupObj = bs4.BeautifulSoup(f, "lxml")

    df = pd.DataFrame([(x["uri"], *x["t"].split("T"), x["u"], x["desc"])
                       for x in soupObj.find_all("log")],
                      columns=["Database", "Date", "Time", "User", "Description"])
    df.to_excel("logfile.xlsx", index=False)

Expected output is to get the final excel sheet of all logfile.txt from all logfolders
But I am get only the data from last logfile.txt with is logfolder_4 in the final excel sheet with is logfile.xlsx what I am doing wrong.

Asked By: Rejoy

||

Source

Answer 1

I believe you need pandas.concat.

Replace the last for-loop bloc by this :

dfList = []
for name in filelist:
    with open(name,"r") as f:
        soupObj = bs4.BeautifulSoup(f, "lxml")

    df = pd.DataFrame([(x["uri"], *x["t"].split("T"), x["u"], x["desc"])
                       for x in soupObj.find_all("log")],
                      columns=["Database", "Date", "Time", "User", "Description"])
    
    dfList.append(df)

pd.concat(dfList).to_excel("logfile.xlsx", index=False)

Answered By: Timeless

How to append data to excel into single excel sheet using for loop

Question:

Answers: