Where to use yield in Python best?
Question:
I know how yield works. I know permutation, think it just as a math simplicity.
But what’s yield‘s true force? When should I use it? A simple and good example is better.
Answers:
yield is best used when you have a function that returns a sequence and you want to iterate over that sequence, but you do not need to have every value in memory at once.
For example, I have a python script that parses a large list of CSV files, and I want to return each line to be processed in another function. I don’t want to store the megabytes of data in memory all at once, so I yield each line in a python data structure. So the function to get lines from the file might look something like:
def get_lines(files):
for f in files:
for line in f:
#preprocess line
yield line
I can then use the same syntax as with lists to access the output of this function:
for line in get_lines(files):
#process line
but I save a lot of memory usage.
Another use is in a network client. Use ‘yield’ in a generator function to round-robin through multiple sockets without the complexity of threads.
For example, I had a hardware test client that needed to send a R,G,B planes of an image to firmware. The data needed to be sent in lockstep: red, green, blue, red, green, blue. Rather than spawn three threads, I had a generator that read from the file, encoded the buffer. Each buffer was a ‘yield buf’. End of file, function returned and I had end-of-iteration.
My client code looped through the three generator functions, getting buffers until end-of-iteration.
Simply put, yield gives you a generator. You’d use it where you would normally use a return in a function. As a really contrived example cut and pasted from a prompt…
>>> def get_odd_numbers(i):
... return range(1, i, 2)
...
>>> def yield_odd_numbers(i):
... for x in range(1, i, 2):
... yield x
...
>>> foo = get_odd_numbers(10)
>>> bar = yield_odd_numbers(10)
>>> foo
[1, 3, 5, 7, 9]
>>> bar
<generator object yield_odd_numbers at 0x1029c6f50>
>>> next(bar)
1
>>> next(bar)
3
>>> next(bar)
5
As you can see, in the first case foo holds the entire list in memory at once. It’s not a big deal for a list with 5 elements, but what if you want a list of 5 million? Not only is this a huge memory eater, it also costs a lot of time to build at the time that the function is called. In the second case, bar just gives you a generator. A generator is an iterable–which means you can use it in a for loop, etc, but each value can only be accessed once. All the values are also not stored in memory at the same time; the generator object "remembers" where it was in the looping the last time you called it–this way, if you’re using an iterable to (say) count to 50 billion, you don’t have to count to 50 billion all at once and store the 50 billion numbers to count through. Again, this is a pretty contrived example, you probably would use itertools if you really wanted to count to 50 billion. 🙂
This is the most simple use case of generators. As you said, it can be used to write efficient permutations, using yield to push things up through the call stack instead of using some sort of stack variable. Generators can also be used for specialized tree traversal, and all manner of other things.
Further reading:
- python wiki http://wiki.python.org/moin/Generators
- PEP on generators http://www.python.org/dev/peps/pep-0255/
I’m reading Data Structures and Algorithms in Python
There is a fibonacci function using yield. I think it’s the best moment to use yield.
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a+b
you can use this like:
gen = fibonacci()
for i, f in enumerate(gen):
print(i, f)
if i >= 100: break
So, I think, maybe, when the next element is depending on previous elements, e.g., digital filters, it’s time to use yield.
I know how yield works. I know permutation, think it just as a math simplicity.
But what’s yield‘s true force? When should I use it? A simple and good example is better.
yield is best used when you have a function that returns a sequence and you want to iterate over that sequence, but you do not need to have every value in memory at once.
For example, I have a python script that parses a large list of CSV files, and I want to return each line to be processed in another function. I don’t want to store the megabytes of data in memory all at once, so I yield each line in a python data structure. So the function to get lines from the file might look something like:
def get_lines(files):
for f in files:
for line in f:
#preprocess line
yield line
I can then use the same syntax as with lists to access the output of this function:
for line in get_lines(files):
#process line
but I save a lot of memory usage.
Another use is in a network client. Use ‘yield’ in a generator function to round-robin through multiple sockets without the complexity of threads.
For example, I had a hardware test client that needed to send a R,G,B planes of an image to firmware. The data needed to be sent in lockstep: red, green, blue, red, green, blue. Rather than spawn three threads, I had a generator that read from the file, encoded the buffer. Each buffer was a ‘yield buf’. End of file, function returned and I had end-of-iteration.
My client code looped through the three generator functions, getting buffers until end-of-iteration.
Simply put, yield gives you a generator. You’d use it where you would normally use a return in a function. As a really contrived example cut and pasted from a prompt…
>>> def get_odd_numbers(i):
... return range(1, i, 2)
...
>>> def yield_odd_numbers(i):
... for x in range(1, i, 2):
... yield x
...
>>> foo = get_odd_numbers(10)
>>> bar = yield_odd_numbers(10)
>>> foo
[1, 3, 5, 7, 9]
>>> bar
<generator object yield_odd_numbers at 0x1029c6f50>
>>> next(bar)
1
>>> next(bar)
3
>>> next(bar)
5
As you can see, in the first case foo holds the entire list in memory at once. It’s not a big deal for a list with 5 elements, but what if you want a list of 5 million? Not only is this a huge memory eater, it also costs a lot of time to build at the time that the function is called. In the second case, bar just gives you a generator. A generator is an iterable–which means you can use it in a for loop, etc, but each value can only be accessed once. All the values are also not stored in memory at the same time; the generator object "remembers" where it was in the looping the last time you called it–this way, if you’re using an iterable to (say) count to 50 billion, you don’t have to count to 50 billion all at once and store the 50 billion numbers to count through. Again, this is a pretty contrived example, you probably would use itertools if you really wanted to count to 50 billion. 🙂
This is the most simple use case of generators. As you said, it can be used to write efficient permutations, using yield to push things up through the call stack instead of using some sort of stack variable. Generators can also be used for specialized tree traversal, and all manner of other things.
Further reading:
- python wiki http://wiki.python.org/moin/Generators
- PEP on generators http://www.python.org/dev/peps/pep-0255/
I’m reading Data Structures and Algorithms in Python
There is a fibonacci function using yield. I think it’s the best moment to use yield.
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a+b
you can use this like:
gen = fibonacci()
for i, f in enumerate(gen):
print(i, f)
if i >= 100: break
So, I think, maybe, when the next element is depending on previous elements, e.g., digital filters, it’s time to use yield.