An elegant and fast way to consecutively iterate over two or more containers in Python?
Question:
I have three collection.deques and what I need to do is to iterate over each of them and perform the same action:
for obj in deque1:
some_action(obj)
for obj in deque2:
some_action(obj)
for obj in deque3:
some_action(obj)
I’m looking for some function XXX which would ideally allow me to write:
for obj in XXX(deque1, deque2, deque3):
some_action(obj)
The important thing here is that XXX have to be efficient enough – without making copy or silently using range(), etc. I was expecting to find it in built-in functions, but I found nothing similar to it so far.
Is there such thing already in Python or I have to write a function for that by myself?
Answers:
The answer is in itertools
itertools.chain(*iterables)
Make an iterator that returns elements from the first iterable until
it is exhausted, then proceeds to the
next iterable, until all of the
iterables are exhausted. Used for
treating consecutive sequences as a
single sequence. Equivalent to:
def chain(*iterables):
# chain('ABC', 'DEF') --> A B C D E F
for it in iterables:
for element in it:
yield element
Depending on what order you want to process the items:
import itertools
for items in itertools.izip(deque1, deque2, deque3):
for item in items:
some_action(item)
for item in itertools.chain(deque1, deque2, deque3):
some_action(item)
I’d recommend doing this to avoid hard-coding the actual deques or number of deques:
deques = [deque1, deque2, deque3]
for item in itertools.chain(*deques):
some_action(item)
To demonstrate the difference in order of the above methods:
>>> a = range(5)
>>> b = range(5)
>>> c = range(5)
>>> d = [a, b, c]
>>>
>>> for items in itertools.izip(*d):
... for item in items:
... print item,
...
0 0 0 1 1 1 2 2 2 3 3 3 4 4 4
>>>
>>> for item in itertools.chain(*d):
... print item,
...
0 1 2 3 4 0 1 2 3 4 0 1 2 3 4
>>>
How about zip?
for obj in zip(deque1, deque2, deque3):
for sub_obj in obj:
some_action(sub_obj)
Use itertools.chain(deque1, deque2, deque3)
Accepts a bunch of iterables, and yields the contents for each of them in sequence.
def XXX(*lists):
for aList in lists:
for item in aList:
yield item
l1 = [1, 2, 3, 4]
l2 = ['a', 'b', 'c']
l3 = [1.0, 1.1, 1.2]
for item in XXX(l1, l2, l3):
print item
1
2
3
4
a
b
c
1.0
1.1
1.2
It looks like you want itertools.chain:
“Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted. Used for treating consecutive sequences as a single sequence.”
If I understand your question correctly, then you can use map
with the first argument set to None, and all the other arguments as your lists to iterate over.
E.g (from an iPython prompt, but you get the idea):
In [85]: p = [1,2,3,4]
In [86]: q = ['a','b','c','d']
In [87]: f = ['Hi', 'there', 'world', '.']
In [88]: for i,j,k in map(None, p,q,f):
....: print i,j,k
....:
....:
1 a Hi
2 b there
3 c world
4 d .
Call me crazy, but why is using itertools thought to be necessary? What’s wrong with:
def perform_func_on_each_object_in_each_of_multiple_containers(func, containers):
for container in containers:
for obj in container:
func(obj)
perform_func_on_each_object_in_each_of_multiple_containers(some_action, (deque1, deque2, deque3)
Even crazier: you probably are going to use this once. Why not just do:
for d in (deque1, deque2, deque3):
for obj in d:
some_action(obj)
What’s going on there is immediately obvious without having to look at the code/docs for the long-name function or having to look up the docs for itertools.something()
I would simply do this :
for obj in deque1 + deque2 + deque3:
some_action(obj)
>>> a = [[],[],[]]
>>> b = [[],[],[]]
>>> for c in [*a,*b]:
c.append("derp")
>>> a
[['derp'], ['derp'], ['derp']]
>>> b
[['derp'], ['derp'], ['derp']]
>>>
I have three collection.deques and what I need to do is to iterate over each of them and perform the same action:
for obj in deque1:
some_action(obj)
for obj in deque2:
some_action(obj)
for obj in deque3:
some_action(obj)
I’m looking for some function XXX which would ideally allow me to write:
for obj in XXX(deque1, deque2, deque3):
some_action(obj)
The important thing here is that XXX have to be efficient enough – without making copy or silently using range(), etc. I was expecting to find it in built-in functions, but I found nothing similar to it so far.
Is there such thing already in Python or I have to write a function for that by myself?
The answer is in itertools
itertools.chain(*iterables)
Make an iterator that returns elements from the first iterable until
it is exhausted, then proceeds to the
next iterable, until all of the
iterables are exhausted. Used for
treating consecutive sequences as a
single sequence. Equivalent to:def chain(*iterables): # chain('ABC', 'DEF') --> A B C D E F for it in iterables: for element in it: yield element
Depending on what order you want to process the items:
import itertools
for items in itertools.izip(deque1, deque2, deque3):
for item in items:
some_action(item)
for item in itertools.chain(deque1, deque2, deque3):
some_action(item)
I’d recommend doing this to avoid hard-coding the actual deques or number of deques:
deques = [deque1, deque2, deque3]
for item in itertools.chain(*deques):
some_action(item)
To demonstrate the difference in order of the above methods:
>>> a = range(5)
>>> b = range(5)
>>> c = range(5)
>>> d = [a, b, c]
>>>
>>> for items in itertools.izip(*d):
... for item in items:
... print item,
...
0 0 0 1 1 1 2 2 2 3 3 3 4 4 4
>>>
>>> for item in itertools.chain(*d):
... print item,
...
0 1 2 3 4 0 1 2 3 4 0 1 2 3 4
>>>
How about zip?
for obj in zip(deque1, deque2, deque3):
for sub_obj in obj:
some_action(sub_obj)
Use itertools.chain(deque1, deque2, deque3)
Accepts a bunch of iterables, and yields the contents for each of them in sequence.
def XXX(*lists):
for aList in lists:
for item in aList:
yield item
l1 = [1, 2, 3, 4]
l2 = ['a', 'b', 'c']
l3 = [1.0, 1.1, 1.2]
for item in XXX(l1, l2, l3):
print item
1
2
3
4
a
b
c
1.0
1.1
1.2
It looks like you want itertools.chain:
“Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted. Used for treating consecutive sequences as a single sequence.”
If I understand your question correctly, then you can use map
with the first argument set to None, and all the other arguments as your lists to iterate over.
E.g (from an iPython prompt, but you get the idea):
In [85]: p = [1,2,3,4]
In [86]: q = ['a','b','c','d']
In [87]: f = ['Hi', 'there', 'world', '.']
In [88]: for i,j,k in map(None, p,q,f):
....: print i,j,k
....:
....:
1 a Hi
2 b there
3 c world
4 d .
Call me crazy, but why is using itertools thought to be necessary? What’s wrong with:
def perform_func_on_each_object_in_each_of_multiple_containers(func, containers):
for container in containers:
for obj in container:
func(obj)
perform_func_on_each_object_in_each_of_multiple_containers(some_action, (deque1, deque2, deque3)
Even crazier: you probably are going to use this once. Why not just do:
for d in (deque1, deque2, deque3):
for obj in d:
some_action(obj)
What’s going on there is immediately obvious without having to look at the code/docs for the long-name function or having to look up the docs for itertools.something()
I would simply do this :
for obj in deque1 + deque2 + deque3:
some_action(obj)
>>> a = [[],[],[]]
>>> b = [[],[],[]]
>>> for c in [*a,*b]:
c.append("derp")
>>> a
[['derp'], ['derp'], ['derp']]
>>> b
[['derp'], ['derp'], ['derp']]
>>>