Read from a log file as it's being written using python

Question

I’m trying to find a nice way to read a log file in real time using python. I’d like to process lines from a log file one at a time as it is written. Somehow I need to keep trying to read the file until it is created and then continue to process lines until I terminate the process. Is there an appropriate way to do this? Thanks.

Asked By: Anon

||

Source

Answer 1

Maybe you could do a system call to

tail -f

using os.system()

Answered By: Guillaume Lebourgeois

Answer 2

Take a look at this PDF starting at page 38, ~slide I-77 and you’ll find all the info you need. Of course the rest of the slides are amazing, too, but those specifically deal with your issue:

import time
def follow(thefile):
    thefile.seek(0,2) # Go to the end of the file
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1) # Sleep briefly
            continue
        yield line

Answered By: Wayne Werner

Answer 3

You could try with something like this:

import time

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        print line, # already has newline

Example was extracted from here.

Answered By: Pablo Santa Cruz

Answer 4

As this is Python and logging tagged, there is another possibility to do this.

I assume this is based on a Python logger, logging.Handler based.

You can just create a class that gets the (named) logger instance and overwrite the emit function to put it onto a GUI (if you need console just add a console handler to the file handler)

Example:

import logging

class log_viewer(logging.Handler):
    """ Class to redistribute python logging data """

    # have a class member to store the existing logger
    logger_instance = logging.getLogger("SomeNameOfYourExistingLogger")

    def __init__(self, *args, **kwargs):
         # Initialize the Handler
         logging.Handler.__init__(self, *args)

         # optional take format
         # setFormatter function is derived from logging.Handler 
         for key, value in kwargs.items():
             if "{}".format(key) == "format":
                 self.setFormatter(value)

         # make the logger send data to this class
         self.logger_instance.addHandler(self)

    def emit(self, record):
        """ Overload of logging.Handler method """

        record = self.format(record)

        # ---------------------------------------
        # Now you can send it to a GUI or similar
        # "Do work" starts here.
        # ---------------------------------------

        # just as an example what e.g. a console
        # handler would do:
        print(record)

I am currently using similar code to add a TkinterTreectrl.Multilistbox for viewing logger output at runtime.

Off-Side: The logger only gets data as soon as it is initialized, so if you want to have all your data available, you need to initialize it at the very beginning. (I know this is what is expected, but I think it is worth being mentioned.)

Answered By: R4PH43L

Read from a log file as it's being written using python

Question:

Answers: