Redirect print command in python script through tqdm.write()
Question:
I’m using tqdm in Python to display console-progressbars in our scripts.
However, I have to call functions which print messages to the console as well and which I can’t change.
In general, writing to the console while displaying progress bars in the console messes up the display like so:
from time import sleep
from tqdm import tqdm
def blabla():
print "Foo blabla"
for k in tqdm(range(3)):
blabla()
sleep(.5)
This creates the output:
0%| | 0/3 [00:00<?, ?it/s]Foo
blabla
33%|###########6 | 1/3 [00:00<00:01, 2.00it/s]Foo
blabla
67%|#######################3 | 2/3 [00:01<00:00, 2.00it/s]Foo
blabla
100%|###################################| 3/3 [00:01<00:00, 2.00it/s]
According to the documentation of tqdm the method tqdm.write() provides a means to write messages to the console without breaking the displayed progressbars.
Thus, the right output is provided by this snippet:
from time import sleep
from tqdm import tqdm
def blabla():
tqdm.write("Foo blabla")
for k in tqdm(range(3)):
blabla()
sleep(.5)
And looks like this:
Foo blabla
Foo blabla
Foo blabla
100%|###################################| 3/3 [00:01<00:00, 1.99it/s]
On the other hand, there is this solution which permits to silence those functions by quite elegantly redirecting sys.stdout into the void.
This works perfectly well for silencing the functions.
Since I want to display the messages from these functions nonetheless without breaking the progress bars, I tried to merge both solutions into one by redirecting sys.stdout to tqdm.write() and, in turn, letting tqdm.write() write to the old sys.stdout.
This results in the snippet:
from time import sleep
import contextlib
import sys
from tqdm import tqdm
class DummyFile(object):
file = None
def __init__(self, file):
self.file = file
def write(self, x):
tqdm.write(x, file=self.file)
@contextlib.contextmanager
def nostdout():
save_stdout = sys.stdout
sys.stdout = DummyFile(save_stdout)
yield
sys.stdout = save_stdout
def blabla():
print "Foo blabla"
for k in tqdm(range(3)):
with nostdout():
blabla()
sleep(.5)
However, this actually creates an even more messed up output as before:
0%| | 0/3 [00:00<?, ?it/s]Foo
blabla
33%|###########6 | 1/3 [00:00<00:01, 2.00it/s]Foo
blabla
67%|#######################3 | 2/3 [00:01<00:00, 2.00it/s]Foo
blabla
100%|###################################| 3/3 [00:01<00:00, 2.00it/s]
FYI: calling tqdm.write(..., end="") inside DummyFile.write() creates the same result as the first output which is still messed up.
I can’t understand why this wouldn’t work, since tqdm.write() is supposed to manage clearing the progress bar before writing the message and then rewriting the progress bar.
What am I missing?
Answers:
Redirecting sys.stdout is always tricky, and it becomes a nightmare when two applications are twiddling with it at the same time.
Here the trick is that tqdm by default prints to sys.stderr, not sys.stdout. Normally, tqdm has an anti-mixup strategy for these two special channels, but since you are redirecting sys.stdout, tqdm gets confused because the file handle changes.
Thus, you just need to explicitly specify file=sys.stdout to tqdm and it will work:
from time import sleep
import contextlib
import sys
from tqdm import tqdm
class DummyFile(object):
file = None
def __init__(self, file):
self.file = file
def write(self, x):
# Avoid print() second call (useless n)
if len(x.rstrip()) > 0:
tqdm.write(x, file=self.file)
@contextlib.contextmanager
def nostdout():
save_stdout = sys.stdout
sys.stdout = DummyFile(sys.stdout)
yield
sys.stdout = save_stdout
def blabla():
print("Foo blabla")
# tqdm call to sys.stdout must be done BEFORE stdout redirection
# and you need to specify sys.stdout, not sys.stderr (default)
for _ in tqdm(range(3), file=sys.stdout):
with nostdout():
blabla()
sleep(.5)
print('Done!')
I also added a few more tricks to make the output nicer (eg, no useless n when using print() without end='').
/EDIT: in fact it seems you can do the stdout redirection after starting tqdm, you just need to specify dynamic_ncols=True in tqdm.
It might be the bad way, but I change built-in print function.
import inspect
import tqdm
# store builtin print
old_print = print
def new_print(*args, **kwargs):
# if tqdm.tqdm.write raises error, use builtin print
try:
tqdm.tqdm.write(*args, **kwargs)
except:
old_print(*args, ** kwargs)
# globaly replace print with new_print
inspect.builtins.print = new_print
By mixing, user493630 and gaborous answers, I created this context manager which avoid having to use the file=sys.stdout parameter of tqdm.
import inspect
import contextlib
import tqdm
@contextlib.contextmanager
def redirect_to_tqdm():
# Store builtin print
old_print = print
def new_print(*args, **kwargs):
# If tqdm.tqdm.write raises error, use builtin print
try:
tqdm.tqdm.write(*args, **kwargs)
except:
old_print(*args, ** kwargs)
try:
# Globaly replace print with new_print
inspect.builtins.print = new_print
yield
finally:
inspect.builtins.print = old_print
To use it, simply:
for i in tqdm.tqdm(range(100)):
with redirect_to_tqdm():
time.sleep(.1)
print(i)
To simplify even further, it is possible to wrap the code in a new function:
def tqdm_redirect(*args, **kwargs):
with redirect_to_tqdm():
for x in tqdm.tqdm(*args, **kwargs):
yield x
for i in tqdm_redirect(range(20)):
time.sleep(.1)
print(i)
OP’s solution is almost correct. The test in tqdm library that messes up your output is this one (https://github.com/tqdm/tqdm/blob/master/tqdm/_tqdm.py#L546-L549):
if hasattr(inst, "start_t") and (inst.fp == fp or all(
f in (sys.stdout, sys.stderr) for f in (fp, inst.
inst.clear(nolock=True)
inst_cleared.append(inst)
tqdm.write is testing the file you provided to see if there is risk of collision between the text to print and potential tqdm bars. In your case stdout and stderr get mixed in the terminal so there is a collision. To counter this, when the test passes, tqdm clears the bars, prints the text and draws the bars back after.
Here, the test fp == sys.stdout fails because sys.stdout became DummyFile, and fp is the real sys.stdout, so the cleaning behavior is not enabled. A simple equality operator in DummyFile fixes everything.
class DummyFile(object):
def __init__(self, file):
self.file = file
def write(self, x):
tqdm.write(x, end="", file=self.file)
def __eq__(self, other):
return other is self.file
Also, since print passes a newline to sys.stdout (or not depending on the user choice), you don’t want tqdm to add another one on its own, so it’s better to set the option end='' than performing strip on the content.
Advantages of this solution
With gaborous’s answer, tqdm(..., file=sys.stdout) pollutes your output stream with pieces of bar. By keeping file=sys.stdout (default), you keep your streams separated.
With Conchylicultor and user493630’s answers, you only patch print. However other systems such as logging directly stream to sys.stdout, so they don’t go through tqdm.write.
I used the solution of @gaborous (https://stackoverflow.com/a/37243211/17419428 – Best answer so far). It worked like a charm for print instructions with just one argument. However, in my case, it introduced newlines characters when using print instructions with several arguments, such as:
print('1','2','3')
1
2
3
The solution (after reading and trying several alternatives) was to use *args and **kwargs arguments. My complete solution is (which I now notice it is very similar to https://stackoverflow.com/a/42424890/17419428):
class DummyFile(object):
file = None
def __init__(self, file):
self.file = file
def write(self, *args, **kwargs):
kwargs['file'] = self.file
kwargs['end'] = ''
# I had to print each argument separately, with kwargs['end'] = ''
for i in args:
tqdm.write(str(i), **kwargs)
def __eq__(self, other):
return other is self.file
def flush(self):
pass
@contextlib.contextmanager
def nostdout():
save_stdout = sys.stdout
sys.stdout = DummyFile(sys.stdout)
yield
sys.stdout = save_stdout
def blabla():
print("Foo blabla", "same line")
# tqdm call to sys.stdout must be done BEFORE stdout redirection
# and you need to specify sys.stdout, not sys.stderr (default)
for _ in tqdm(range(3), file=sys.stdout):
with nostdout():
blabla()
sleep(.5)
I’m using tqdm in Python to display console-progressbars in our scripts.
However, I have to call functions which print messages to the console as well and which I can’t change.
In general, writing to the console while displaying progress bars in the console messes up the display like so:
from time import sleep
from tqdm import tqdm
def blabla():
print "Foo blabla"
for k in tqdm(range(3)):
blabla()
sleep(.5)
This creates the output:
0%| | 0/3 [00:00<?, ?it/s]Foo
blabla
33%|###########6 | 1/3 [00:00<00:01, 2.00it/s]Foo
blabla
67%|#######################3 | 2/3 [00:01<00:00, 2.00it/s]Foo
blabla
100%|###################################| 3/3 [00:01<00:00, 2.00it/s]
According to the documentation of tqdm the method tqdm.write() provides a means to write messages to the console without breaking the displayed progressbars.
Thus, the right output is provided by this snippet:
from time import sleep
from tqdm import tqdm
def blabla():
tqdm.write("Foo blabla")
for k in tqdm(range(3)):
blabla()
sleep(.5)
And looks like this:
Foo blabla
Foo blabla
Foo blabla
100%|###################################| 3/3 [00:01<00:00, 1.99it/s]
On the other hand, there is this solution which permits to silence those functions by quite elegantly redirecting sys.stdout into the void.
This works perfectly well for silencing the functions.
Since I want to display the messages from these functions nonetheless without breaking the progress bars, I tried to merge both solutions into one by redirecting sys.stdout to tqdm.write() and, in turn, letting tqdm.write() write to the old sys.stdout.
This results in the snippet:
from time import sleep
import contextlib
import sys
from tqdm import tqdm
class DummyFile(object):
file = None
def __init__(self, file):
self.file = file
def write(self, x):
tqdm.write(x, file=self.file)
@contextlib.contextmanager
def nostdout():
save_stdout = sys.stdout
sys.stdout = DummyFile(save_stdout)
yield
sys.stdout = save_stdout
def blabla():
print "Foo blabla"
for k in tqdm(range(3)):
with nostdout():
blabla()
sleep(.5)
However, this actually creates an even more messed up output as before:
0%| | 0/3 [00:00<?, ?it/s]Foo
blabla
33%|###########6 | 1/3 [00:00<00:01, 2.00it/s]Foo
blabla
67%|#######################3 | 2/3 [00:01<00:00, 2.00it/s]Foo
blabla
100%|###################################| 3/3 [00:01<00:00, 2.00it/s]
FYI: calling tqdm.write(..., end="") inside DummyFile.write() creates the same result as the first output which is still messed up.
I can’t understand why this wouldn’t work, since tqdm.write() is supposed to manage clearing the progress bar before writing the message and then rewriting the progress bar.
What am I missing?
Redirecting sys.stdout is always tricky, and it becomes a nightmare when two applications are twiddling with it at the same time.
Here the trick is that tqdm by default prints to sys.stderr, not sys.stdout. Normally, tqdm has an anti-mixup strategy for these two special channels, but since you are redirecting sys.stdout, tqdm gets confused because the file handle changes.
Thus, you just need to explicitly specify file=sys.stdout to tqdm and it will work:
from time import sleep
import contextlib
import sys
from tqdm import tqdm
class DummyFile(object):
file = None
def __init__(self, file):
self.file = file
def write(self, x):
# Avoid print() second call (useless n)
if len(x.rstrip()) > 0:
tqdm.write(x, file=self.file)
@contextlib.contextmanager
def nostdout():
save_stdout = sys.stdout
sys.stdout = DummyFile(sys.stdout)
yield
sys.stdout = save_stdout
def blabla():
print("Foo blabla")
# tqdm call to sys.stdout must be done BEFORE stdout redirection
# and you need to specify sys.stdout, not sys.stderr (default)
for _ in tqdm(range(3), file=sys.stdout):
with nostdout():
blabla()
sleep(.5)
print('Done!')
I also added a few more tricks to make the output nicer (eg, no useless n when using print() without end='').
/EDIT: in fact it seems you can do the stdout redirection after starting tqdm, you just need to specify dynamic_ncols=True in tqdm.
It might be the bad way, but I change built-in print function.
import inspect
import tqdm
# store builtin print
old_print = print
def new_print(*args, **kwargs):
# if tqdm.tqdm.write raises error, use builtin print
try:
tqdm.tqdm.write(*args, **kwargs)
except:
old_print(*args, ** kwargs)
# globaly replace print with new_print
inspect.builtins.print = new_print
By mixing, user493630 and gaborous answers, I created this context manager which avoid having to use the file=sys.stdout parameter of tqdm.
import inspect
import contextlib
import tqdm
@contextlib.contextmanager
def redirect_to_tqdm():
# Store builtin print
old_print = print
def new_print(*args, **kwargs):
# If tqdm.tqdm.write raises error, use builtin print
try:
tqdm.tqdm.write(*args, **kwargs)
except:
old_print(*args, ** kwargs)
try:
# Globaly replace print with new_print
inspect.builtins.print = new_print
yield
finally:
inspect.builtins.print = old_print
To use it, simply:
for i in tqdm.tqdm(range(100)):
with redirect_to_tqdm():
time.sleep(.1)
print(i)
To simplify even further, it is possible to wrap the code in a new function:
def tqdm_redirect(*args, **kwargs):
with redirect_to_tqdm():
for x in tqdm.tqdm(*args, **kwargs):
yield x
for i in tqdm_redirect(range(20)):
time.sleep(.1)
print(i)
OP’s solution is almost correct. The test in tqdm library that messes up your output is this one (https://github.com/tqdm/tqdm/blob/master/tqdm/_tqdm.py#L546-L549):
if hasattr(inst, "start_t") and (inst.fp == fp or all(
f in (sys.stdout, sys.stderr) for f in (fp, inst.
inst.clear(nolock=True)
inst_cleared.append(inst)
tqdm.write is testing the file you provided to see if there is risk of collision between the text to print and potential tqdm bars. In your case stdout and stderr get mixed in the terminal so there is a collision. To counter this, when the test passes, tqdm clears the bars, prints the text and draws the bars back after.
Here, the test fp == sys.stdout fails because sys.stdout became DummyFile, and fp is the real sys.stdout, so the cleaning behavior is not enabled. A simple equality operator in DummyFile fixes everything.
class DummyFile(object):
def __init__(self, file):
self.file = file
def write(self, x):
tqdm.write(x, end="", file=self.file)
def __eq__(self, other):
return other is self.file
Also, since print passes a newline to sys.stdout (or not depending on the user choice), you don’t want tqdm to add another one on its own, so it’s better to set the option end='' than performing strip on the content.
Advantages of this solution
With gaborous’s answer, tqdm(..., file=sys.stdout) pollutes your output stream with pieces of bar. By keeping file=sys.stdout (default), you keep your streams separated.
With Conchylicultor and user493630’s answers, you only patch print. However other systems such as logging directly stream to sys.stdout, so they don’t go through tqdm.write.
I used the solution of @gaborous (https://stackoverflow.com/a/37243211/17419428 – Best answer so far). It worked like a charm for print instructions with just one argument. However, in my case, it introduced newlines characters when using print instructions with several arguments, such as:
print('1','2','3')
1
2
3
The solution (after reading and trying several alternatives) was to use *args and **kwargs arguments. My complete solution is (which I now notice it is very similar to https://stackoverflow.com/a/42424890/17419428):
class DummyFile(object):
file = None
def __init__(self, file):
self.file = file
def write(self, *args, **kwargs):
kwargs['file'] = self.file
kwargs['end'] = ''
# I had to print each argument separately, with kwargs['end'] = ''
for i in args:
tqdm.write(str(i), **kwargs)
def __eq__(self, other):
return other is self.file
def flush(self):
pass
@contextlib.contextmanager
def nostdout():
save_stdout = sys.stdout
sys.stdout = DummyFile(sys.stdout)
yield
sys.stdout = save_stdout
def blabla():
print("Foo blabla", "same line")
# tqdm call to sys.stdout must be done BEFORE stdout redirection
# and you need to specify sys.stdout, not sys.stderr (default)
for _ in tqdm(range(3), file=sys.stdout):
with nostdout():
blabla()
sleep(.5)