How to read a file (or stdin) line by line in Python not waiting for reading entire file

Question

I want to read and process some large files line by line in python and output results in the terminal. I have gone through How do I read from stdin? and How do I write a unix filter in python?, but I am looking for methods which do not wait till the entire file is read into memory.

I would be using both of these commands:

cat fileName | python myScript1.py
python myScript2.py fileName

Asked By: BiGYaN

||

Source

Answer 1

Just iterate over the file:

with open('huge.file') as hf:
  for line in hf:
    if 'important' in line:
      print(line)

This will require O(1) memory.

To read from stdin, simply iterate over sys.stdin instead of hf:

import sys
for line in sys.stdin:
  if 'important' in line:
    print(line)

Answered By: phihag

Answer 2

This is the standard behavior of file objects in Python:

with open("myfile.txt", "r") as myfile:
    for line in myfile:
        # do something with the current line

or

for line in sys.stdin:
    # do something with the current line

Answered By: Tim Pietzcker

Answer 3

if __name__ == '__main__':
    while 1:
        try:
            a=raw_input()
        except EOFError:
            break
        print a

This will read from stdin til EOF.
To read a file using the second method, you can use Tim’s method

i.e.

with open("myfile.txt", "r") as myfile:
    for line in myfile:
        print line
        # do something with the current line

Answered By: spicavigo

How to read a file (or stdin) line by line in Python not waiting for reading entire file

Question:

Answers: