Making tqdm write to log files

Question

tqdm is a nice python library to keep track of progress through an iterable.

It’s default mode of operation is to repeatedly clear a line and redraw with a carriage but this produced quite nasty output when combined with logging. Is there a way I can get this to write to log files periodically rather than using this print?

This is the best I’ve got is my own hacky implementation:

def my_tqdm(iterable):
    "Like tqdm but us logging. Include estimated time and time taken."
    start = time.time()
    for i, item in enumerate(iterable):
        elapsed = time.time() - start
        rate = elapsed / (i + 1)
        estimated = rate * len(iterable) - elapsed
        num_items = len(iterable)
        LOGGER.info(
            "Processed %d of %d items (%.1f%%) in %.1fs (%.1fs remaining, %.1f s/item)",
            i,
            num_items,
            i / num_items * 100,
            elapsed,
            estimated,
            rate,
        )
        yield item

But it’d be better if I could do this with tqdm itself so that people don’t moan at me in code reviews.

Asked By: Att Righ

||

Source

Answer 1

You could redirect the outputs of the TQDM progress bar to a null device (e.g. /dev/null), and manually print/log the status bar whenever you want – either on every iteration, or at a certain interval.

For example:

import os
import time
import logging
from tqdm import tqdm

LOG_INTERVAL = 5

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('tqdm_logger')
progress_bar = tqdm(range(20), file=open(os.devnull, 'w'))

for i in progress_bar:
    # do something meaningful instead...
    time.sleep(0.1)

    if progress_bar.n % LOG_INTERVAL == 0:
        logger.info(str(progress_bar))

This code block will produce the following outputs:

INFO:tqdm_logger:  0%|          | 0/20 [00:00<?, ?it/s]
INFO:tqdm_logger: 25%|██▌       | 5/20 [00:00<00:01,  9.59it/s]
INFO:tqdm_logger: 50%|█████     | 10/20 [00:01<00:01,  9.54it/s]
INFO:tqdm_logger: 75%|███████▌  | 15/20 [00:01<00:00,  9.54it/s]

Answered By: matanb

Making tqdm write to log files

Question:

Answers: