tqdm progressbar and zip built-in do not work together
Question:
tqdm is a Python module to easily print in the console a dynamically updating progressbar. For example
from tqdm import tqdm
from time import sleep
for _ in tqdm(range(10)):
sleep(0.1)
prints a dynamic progressbar in the console for 1sec as the iteration executes:
I have not figured out how to use tqdm with the builtin zip object.
The use case of this would be to iterate over two corresponding lists with a console progressbar.
For example, I would expect this to work:
for _, _ in tqdm(zip(range(10), range(10))):
sleep(0.1)
but the progressbar printed to the console in this case is not correct:
A workaround is to use tqdm with enumerate, however then an iterator index must be defined and managed.
Answers:
For you have a progress-bar is expected that you can predict the length of your data structure.
range
implements the hook method __len__
, so you can discover the length doing built-in len
>>> dir(range(10))
[ '__le__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index', 'start', 'step', 'stop']
>>> len(range(10))
10
zip
, however, does not provide a way to guess the length of the wrapped structure, so probably that’s why because tqdm
can not show the progress bar.
dir(zip(range(10))) # no __len__ here
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> len(zip(range(10)))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: object of type 'zip' has no len()
Edit:
Yeah, that’s it. Look the docs
…
Wrapping enumerated iterables: use enumerate(tqdm(...))
instead of
tqdm(enumerate(...))
. The same applies to numpy.ndenumerate
. This is
because enumerate
functions tend to hide the length of iterables. tqdm
does not.
…
Manual control on tqdm()
updates by using a with
statement:
with tqdm(total=100) as pbar:
for i in range(10):
pbar.update(10)
If the optional variable total (or an iterable with len()
) is
provided, predictive stats are displayed.
with
is also optional (you can just assign tqdm()
to a variable, but
in this case don’t forget to del
or close()
at the end:
pbar = tqdm(total=100)
for i in range(10):
pbar.update(10)
pbar.close()
tqdm
can be used with zip
if a total
keyword argument is provided in the tqdm
call.
The following example demonstrates iteration over corresponding elements in two lists with a working __tqdm__
progress bar for the case where a total
keyword argument is used:
The issue is that tqdm
needs to know the length of the iterable ahead of time. Because zip
is meant to handle iterables with different lengths, it does not have as an attribute a single length of its arguments.
So, __tqdm__
still works nicely with zip
, you just need to provide a little manual control with the total
keyword argument.
Using tqdm>=4.42.0
, you should do:
from tqdm.contrib import tzip
from time import sleep
for _, _ in tzip(range(10), range(10)):
sleep(0.1)
Just to note in https://github.com/tqdm/tqdm#faq-and-known-issues:
- Wrapping generators:
- Generator wrapper functions tend to hide the length of iterables.
tqdm
does not.
- Replace
tqdm(enumerate(...))
with enumerate(tqdm(...))
or tqdm(enumerate(x), total=len(x), ...)
. The same applies to numpy.ndenumerate
.
- Replace
tqdm(zip(a, b))
with zip(tqdm(a), b)
or even zip(tqdm(a), tqdm(b))
.
- The same applies to
itertools
.
- Some useful convenience functions can be found under
tqdm.contrib
.
from tqdm import tqdm
a = range(10)
b = range(10)
for _ in zip(tqdm(a), b):
pass
tqdm is a Python module to easily print in the console a dynamically updating progressbar. For example
from tqdm import tqdm
from time import sleep
for _ in tqdm(range(10)):
sleep(0.1)
prints a dynamic progressbar in the console for 1sec as the iteration executes:
I have not figured out how to use tqdm with the builtin zip object.
The use case of this would be to iterate over two corresponding lists with a console progressbar.
For example, I would expect this to work:
for _, _ in tqdm(zip(range(10), range(10))):
sleep(0.1)
but the progressbar printed to the console in this case is not correct:
A workaround is to use tqdm with enumerate, however then an iterator index must be defined and managed.
For you have a progress-bar is expected that you can predict the length of your data structure.
range
implements the hook method __len__
, so you can discover the length doing built-in len
>>> dir(range(10))
[ '__le__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index', 'start', 'step', 'stop']
>>> len(range(10))
10
zip
, however, does not provide a way to guess the length of the wrapped structure, so probably that’s why because tqdm
can not show the progress bar.
dir(zip(range(10))) # no __len__ here
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> len(zip(range(10)))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: object of type 'zip' has no len()
Edit:
Yeah, that’s it. Look the docs
…
Wrapping enumerated iterables: use
enumerate(tqdm(...))
instead of
tqdm(enumerate(...))
. The same applies tonumpy.ndenumerate
. This is
becauseenumerate
functions tend to hide the length of iterables.tqdm
does not.…
Manual control on
tqdm()
updates by using awith
statement:with tqdm(total=100) as pbar: for i in range(10): pbar.update(10)
If the optional variable total (or an iterable with
len()
) is
provided, predictive stats are displayed.
with
is also optional (you can just assigntqdm()
to a variable, but
in this case don’t forget todel
orclose()
at the end:pbar = tqdm(total=100) for i in range(10): pbar.update(10) pbar.close()
tqdm
can be used with zip
if a total
keyword argument is provided in the tqdm
call.
The following example demonstrates iteration over corresponding elements in two lists with a working __tqdm__
progress bar for the case where a total
keyword argument is used:
The issue is that tqdm
needs to know the length of the iterable ahead of time. Because zip
is meant to handle iterables with different lengths, it does not have as an attribute a single length of its arguments.
So, __tqdm__
still works nicely with zip
, you just need to provide a little manual control with the total
keyword argument.
Using tqdm>=4.42.0
, you should do:
from tqdm.contrib import tzip
from time import sleep
for _, _ in tzip(range(10), range(10)):
sleep(0.1)
Just to note in https://github.com/tqdm/tqdm#faq-and-known-issues:
- Wrapping generators:
- Generator wrapper functions tend to hide the length of iterables.
tqdm
does not.- Replace
tqdm(enumerate(...))
withenumerate(tqdm(...))
ortqdm(enumerate(x), total=len(x), ...)
. The same applies tonumpy.ndenumerate
.- Replace
tqdm(zip(a, b))
withzip(tqdm(a), b)
or evenzip(tqdm(a), tqdm(b))
.- The same applies to
itertools
.- Some useful convenience functions can be found under
tqdm.contrib
.
from tqdm import tqdm
a = range(10)
b = range(10)
for _ in zip(tqdm(a), b):
pass