Read multiple file consecutively

Question

I was trying to read one line ahead of multiple files consecutively, i.e. go to the next file when finish reading the first one. For instance, I have file1 and file2 with content 1 2 3 and 3 4 5 respectively.
I have tried the following; however, it only reads the first two lines of the file and does not go to the next file when it is finished iterating through the first file’s content.

for files in file_list:
        with open(files, 'r') as f:
            line = f.readline()
            next_line = f.readline()

I need to read one line ahead and pass line and next_line to my other functions, so I when I print my line and next line, I want them to be 1,3,3,5 and 2,4 respectively.

For the first file, line=1, next_line=2, line=3 and after iterating through the first file it goes to the next file, where line = 3, next_line=4, line=5…etc

Asked By: User21749

||

Source

Answer 1

You can define a function that will iterate over all the files and yield from these files. Then, you can use this just like you use the file handle returned by open:

def open_multiple_files(*files):
    for file in files:
        with open(file) as f:
            yield from f
# Usage:
for line in open_multiple_files("file1.txt", "file2.txt"):
    print(line.strip())

which will print

You can’t access the next line in the iteration, but you can keep track of the previous line and the current line. Then you can simply name the current line as next_line, the previous line as this_line.

file_handle = open_multiple_files("file1.txt", "file2.txt")
this_line = next(file_handle).strip()

for next_line in file_handle:
    next_line = next_line.strip()
    print("This line: ", this_line, "| Next line: ", next_line)
    this_line = next_line

Which gives:

This line:  1 | Next line:  2
This line:  2 | Next line:  3
This line:  3 | Next line:  3
This line:  3 | Next line:  4
This line:  4 | Next line:  5

If you want to simply read all these files in chunks of two lines, where the first line in the chunk is this_line and the second line is next_line, you can use next inside the loop like so:

for this_line in file_handle:
    this_line = this_line.strip()
    next_line = next(file_handle).strip()
    print("This line: ", this_line, "| Next line: ", next_line)

Since file handles keep track of how far they’ve read, the next call in the first iteration will consume a line, and after the first iteration this_line will be the third line in file_handle, and this will output:

This line:  1 | Next line:  2
This line:  3 | Next line:  3
This line:  4 | Next line:  5

If you don’t want the content of one file to be merged with the other, you can just skip the step that opens multiple files, and move that loop outside the loop above. Of course, in this case your file handle will throw a StopIteration error when there is no next_line, but you can catch that error and set next_line to an empty string (or handle it however else you want):

for file in ["file1.txt", "file2.txt"]:
    with open(file) as f:
        for this_line in f:
            this_line = this_line.strip()
            try:
                next_line = next(f).strip()
            except StopIteration:
                next_line = "<END OF FILE>"
            print("This line: ", this_line, "| Next line: ", next_line)

and this will output:

This line:  1 | Next line:  2
This line:  3 | Next line:  <END OF FILE>
This line:  3 | Next line:  4
This line:  5 | Next line:  <END OF FILE>

Answered By: Pranav Hosangadi

Read multiple file consecutively

Question:

Answers: