List comprehension without [ ] in Python
Question:
Joining a list:
>>> ''.join([ str(_) for _ in xrange(10) ])
'0123456789'
join must take an iterable.
Apparently, join‘s argument is [ str(_) for _ in xrange(10) ], and it’s a list comprehension.
Look at this:
>>>''.join( str(_) for _ in xrange(10) )
'0123456789'
Now, join‘s argument is just str(_) for _ in xrange(10), no [], but the result is the same.
Why? Does str(_) for _ in xrange(10) also produce a list or an iterable?
Answers:
If it’s in parens, but not brackets, it’s technically a generator expression. Generator expressions were first introduced in Python 2.4.
http://wiki.python.org/moin/Generators
The part after the join, ( str(_) for _ in xrange(10) ) is, by itself, a generator expression. You could do something like:
mylist = (str(_) for _ in xrange(10))
''.join(mylist)
and it means exactly the same thing that you wrote in the second case above.
Generators have some very interesting properties, not the least of which is that they don’t end up allocating an entire list when you don’t need one. Instead, a function like join “pumps” the items out of the generator expression one at a time, doing its work on the tiny intermediate parts.
In your particular examples, list and generator probably don’t perform terribly differently, but in general, I prefer using generator expressions (and even generator functions) whenever I can, mostly because it’s extremely rare for a generator to be slower than a full list materialization.
That’s a generator, rather than a list comprehension. Generators are also iterables, but rather than creating the entire list first then passing it to join, it passes each value in the xrange one by one, which can be much more efficient.
The argument to your second join call is a generator expression. It does produce an iterable.
>>>''.join( str(_) for _ in xrange(10) )
This is called a generator expression, and is explained in PEP 289.
The main difference between generator expressions and list comprehensions is that the former don’t create the list in memory.
Note that there’s a third way to write the expression:
''.join(map(str, xrange(10)))
Your second example uses a generator expression rather than a list comprehension. The difference is that with the list comprehension, a list is completely built and passed to .join(). With the generator expression, items are generated one by one and consumed by .join(). The latter uses less memory and is generally faster.
As it happens, the list constructor will happily consume any iterable, including a generator expression. So:
[str(n) for n in xrange(10)]
is just “syntactic sugar” for:
list(str(n) for n in xrange(10))
In other words, a list comprehension is just a generator expression that is turned into a list.
As mentioned it’s a generator expression.
From the documentation:
The parentheses can be omitted on calls with only one argument. See section Calls for the detail.
The other respondents were correct in answering that you had discovered a generator expression (which has a notation similar to list comprehensions but without the surrounding square brackets).
In general, genexps (as they are affectionately known) are more memory efficient and faster than list comprehensions.
HOWEVER, it the case of ''.join(), a list comprehension is both faster and more memory efficient. The reason is that join needs to make two passes over the data, so it actually needs a real list. If you give it one, it can start its work immediately. If you give it a genexp instead, it cannot start work until it builds-up a new list in memory by running the genexp to exhaustion:
~ $ python -m timeit '"".join(str(n) for n in xrange(1000))'
1000 loops, best of 3: 335 usec per loop
~ $ python -m timeit '"".join([str(n) for n in xrange(1000)])'
1000 loops, best of 3: 288 usec per loop
The same result holds when comparing itertools.imap versus map:
~ $ python -m timeit -s'from itertools import imap' '"".join(imap(str, xrange(1000)))'
1000 loops, best of 3: 220 usec per loop
~ $ python -m timeit '"".join(map(str, xrange(1000)))'
1000 loops, best of 3: 212 usec per loop
Joining a list:
>>> ''.join([ str(_) for _ in xrange(10) ])
'0123456789'
join must take an iterable.
Apparently, join‘s argument is [ str(_) for _ in xrange(10) ], and it’s a list comprehension.
Look at this:
>>>''.join( str(_) for _ in xrange(10) )
'0123456789'
Now, join‘s argument is just str(_) for _ in xrange(10), no [], but the result is the same.
Why? Does str(_) for _ in xrange(10) also produce a list or an iterable?
If it’s in parens, but not brackets, it’s technically a generator expression. Generator expressions were first introduced in Python 2.4.
http://wiki.python.org/moin/Generators
The part after the join, ( str(_) for _ in xrange(10) ) is, by itself, a generator expression. You could do something like:
mylist = (str(_) for _ in xrange(10))
''.join(mylist)
and it means exactly the same thing that you wrote in the second case above.
Generators have some very interesting properties, not the least of which is that they don’t end up allocating an entire list when you don’t need one. Instead, a function like join “pumps” the items out of the generator expression one at a time, doing its work on the tiny intermediate parts.
In your particular examples, list and generator probably don’t perform terribly differently, but in general, I prefer using generator expressions (and even generator functions) whenever I can, mostly because it’s extremely rare for a generator to be slower than a full list materialization.
That’s a generator, rather than a list comprehension. Generators are also iterables, but rather than creating the entire list first then passing it to join, it passes each value in the xrange one by one, which can be much more efficient.
The argument to your second join call is a generator expression. It does produce an iterable.
>>>''.join( str(_) for _ in xrange(10) )
This is called a generator expression, and is explained in PEP 289.
The main difference between generator expressions and list comprehensions is that the former don’t create the list in memory.
Note that there’s a third way to write the expression:
''.join(map(str, xrange(10)))
Your second example uses a generator expression rather than a list comprehension. The difference is that with the list comprehension, a list is completely built and passed to .join(). With the generator expression, items are generated one by one and consumed by .join(). The latter uses less memory and is generally faster.
As it happens, the list constructor will happily consume any iterable, including a generator expression. So:
[str(n) for n in xrange(10)]
is just “syntactic sugar” for:
list(str(n) for n in xrange(10))
In other words, a list comprehension is just a generator expression that is turned into a list.
As mentioned it’s a generator expression.
From the documentation:
The parentheses can be omitted on calls with only one argument. See section Calls for the detail.
The other respondents were correct in answering that you had discovered a generator expression (which has a notation similar to list comprehensions but without the surrounding square brackets).
In general, genexps (as they are affectionately known) are more memory efficient and faster than list comprehensions.
HOWEVER, it the case of ''.join(), a list comprehension is both faster and more memory efficient. The reason is that join needs to make two passes over the data, so it actually needs a real list. If you give it one, it can start its work immediately. If you give it a genexp instead, it cannot start work until it builds-up a new list in memory by running the genexp to exhaustion:
~ $ python -m timeit '"".join(str(n) for n in xrange(1000))'
1000 loops, best of 3: 335 usec per loop
~ $ python -m timeit '"".join([str(n) for n in xrange(1000)])'
1000 loops, best of 3: 288 usec per loop
The same result holds when comparing itertools.imap versus map:
~ $ python -m timeit -s'from itertools import imap' '"".join(imap(str, xrange(1000)))'
1000 loops, best of 3: 220 usec per loop
~ $ python -m timeit '"".join(map(str, xrange(1000)))'
1000 loops, best of 3: 212 usec per loop