Output enumerate function to csv in Python

Question

I am trying to understand how to enumerate through a csv file and output the results to another csv file. For example, I have a csv file that has 2 columns, group_id and node_id. I want to use this csv file, loop through it and output the group_id, node_id and a formatted string that mentions the node_id. My code does exactly this when I use the print function however when I attempt to write to another csv file, only the last row is written.

Here is my code:

import csv

with open('input.csv', 'r') as f:

    config_csv = csv.reader(f)

    for row, column in enumerate(config_csv):

        if row == 0:
            continue

        group_id = int(column[0])
        sub_id = column[1]
        node = f"The sub ID for this node is {sub_id}."
        full_output=[group_id, sub_id, node]
        print(full_output)

        with open('output.csv', 'w') as file:
            writer=csv.writer(file)
            writer.writerow(full_output)

csv file (input.csv):

GROUP_ID,SUB_ID
675233,111
877531,222
455632,333

And my print output is:

[675233, 111, 'The sub ID for this node is 111.']
[877531, 222, 'The sub ID for this node is 222.']
[455632, 333, 'The sub ID for this node is 333.']

However my output file (output.csv) only shows the last line:

[455632, 333, 'The sub ID for this node is 333.']

What am I doing wrong? Why doesn’t it output the same to the csv file that I see in the print function?

Asked By: ubernoober

||

Source

Answer 1

As said by the comment of Barmar.

Opening the file in w mode empties the file. Don’t reopen the file in the loop, open it once at the beginning and write a row to it each time through the loop

So I see basically two options.
1- You can use open without "with", and it will be the same result

f = open("demofile.txt", "r")

instead of

with open('input.csv', 'r') as f:

2- You can store all the data you are reading and then write it in the other csv file

For example, you could create a class :

class GroupData :
   group_id = 0
   sub_id = 0
   node = ""

And then create a list of groupdata objets.
Add element in your list while reading the first file.
Then read all the elements in another loop and write in the other file.

Answered By: Lal

Answer 2

Open both files and write the lines as they are processed. It’s not clear from your incorrect indentation but you are probably writing a new file each time through the loop so only end up with the last line.

Also, make sure to use newline='' when opening as per csv documentation:

import csv

# Need Python 3.10 for parenthesized context manager support.
# Use the following one-liner on older versions.
#with open('input.csv', 'r', newline='') as fin, open('output.csv', 'w', newline='') as fout:
with (open('input.csv', 'r', newline='') as fin,
      open('output.csv', 'w', newline='') as fout):

    reader = csv.reader(fin)
    writer = csv.writer(fout)

    # read and copy header to output
    header = next(reader)
    header.append('COMMENT')
    print(header)
    writer.writerow(header)

    for row in reader:
        node = f"The sub ID for this node is {row[1]}."
        row.append(node)
        print(row)
        writer.writerow(row)

Console output:

['GROUP_ID', 'SUB_ID', 'COMMENT']
['675233', '111', 'The sub ID for this node is 111.']
['877531', '222', 'The sub ID for this node is 222.']
['455632', '333', 'The sub ID for this node is 333.']

output.csv:

GROUP_ID,SUB_ID,COMMENT
675233,111,The sub ID for this node is 111.
877531,222,The sub ID for this node is 222.
455632,333,The sub ID for this node is 333.

Answered By: Mark Tolonen

Output enumerate function to csv in Python

Question:

Answers: