Why is this Python generator functioning like this?
Question:
I was looking into this article by RealPython about Python generators. It uses a palindrome generator as a exemplary tool to explain the .send
, .throw
and .close
methods. Building a program following their instructions gives the following code:
def infinite_palindromes():
num = 0
while True:
if is_palindrome(num):
i = (yield num)
if i is not None:
num = i
num += 1
def is_palindrome(num):
# Skip single-digit inputs
if num // 10 == 0:
return False
temp = num
reversed_num = 0
while temp != 0:
reversed_num = (reversed_num * 10) + (temp % 10)
temp = temp // 10
if num == reversed_num:
return True
else:
return False
pal_gen = infinite_palindromes()
for e in pal_gen:
digits = len(str(e))
pal_gen.send(10**(digits))
I watched the flow of the program in debugging and what happens is basically, when the function finds 11
, the first palindrome, it enters into the for e
loop block, the digits
variable is defined, and 10 to the power of that variable value is sent back to the generator. 100 is sent back, and since it is different from None
, num is equated to i and becomes 100, then it’s summed with 1 and becomes 101, the next palindrome. Here is what I don’t get: when that number is throw and correctly identified as a palindrome, the code goes back to the for e in pal_gen:
line, but it doesn’t enter the block. Only the next palindrome, 111
is sent back down and enters the block, where the digits
variable is defined again, and so on and so forth, and this patterns repeats, with only every other palindrome entering the block. Why is that?
Can someone, please, explain to me why that happens?
Answers:
To understand the behaviour, we need to understand how for
loops are implemented in Python. This code:
for e in pal_gen:
digits = len(str(e))
pal_gen.send(10**(digits))
is basically equivalent to (except that no name _iter
is actually generated):
_iter = iter(pal_gen)
while True:
try:
e = next(_iter)
digits = len(str(e))
pal_gen.send(10**(digits))
except StopIteration:
break
However, because pal_gen
is a generator object (created by calling the generator function infinite_palindromes
), it is already an iterator; calling iter
on it returns the same object unchanged. Thus, we effectively have:
while True:
try:
e = next(pal_gen)
digits = len(str(e))
pal_gen.send(10**(digits))
except StopIteration:
break
The other thing to note here is that calling .send
on a generator advances the generator, and so does passing it to next
(i.e., using it like any other iterator). It’s just that the code was written to ignore the value retrieved using .send
.
In fact, next
on a generator is equivalent to calling .send
and passing None
(by convention; Python defines the behaviour this way so that the assignment in the generator code, i = (yield num)
, has something to assign).
So, each time through the loop, two values are sent to the generator:
while True:
try:
e = pal_gen.send(None)
digits = len(str(e))
pal_gen.send(10**(digits))
except StopIteration:
break
The control flow is like so:
None
is sent to the generator. Because it hasn’t yielded yet, this value is discarded (Python requires it to be None
, since it can’t be used by the generator).
- The generator iterates its own loop until
11
is found and yielded. The execution of the generator pauses here: i
is not assigned yet.
- The main loop receives
11
as a value for e
, computes 100
as the next value to work with, and sends that to the generator.
- The generator receives
100
and assigns it to i
, then proceeds with its logic. The next step is if i is not None:
, and that is indeed the case – thus, 100
is assigned to num
. num
is incremented at the end of the loop.
- The loop inside the generator runs into the next iteration, and immediately finds that
101
is a palindrome. Thus, the .send
call returns a value of 101
, but this isn’t assigned anywhere.
- Similarly, in the second iteration of the main loop,
111
will be found and yielded (as the next palindrome after 101
) and assigned to e
in the first call; then 1000
is sent explicitly, causing 1001
to be yielded and ignored. In the third iteration, 1111
is assigned to e
, then 10001
is ignored. In the fourth iteration, 10101
(not 11111
! This is the next palindrome after 10001
) is assigned, and 100001
is ignored. Etc.
If the code is tested at the REPL, the values returned from the explicit .send
will be displayed. This is just a quirk of the REPL.
I was looking into this article by RealPython about Python generators. It uses a palindrome generator as a exemplary tool to explain the .send
, .throw
and .close
methods. Building a program following their instructions gives the following code:
def infinite_palindromes():
num = 0
while True:
if is_palindrome(num):
i = (yield num)
if i is not None:
num = i
num += 1
def is_palindrome(num):
# Skip single-digit inputs
if num // 10 == 0:
return False
temp = num
reversed_num = 0
while temp != 0:
reversed_num = (reversed_num * 10) + (temp % 10)
temp = temp // 10
if num == reversed_num:
return True
else:
return False
pal_gen = infinite_palindromes()
for e in pal_gen:
digits = len(str(e))
pal_gen.send(10**(digits))
I watched the flow of the program in debugging and what happens is basically, when the function finds 11
, the first palindrome, it enters into the for e
loop block, the digits
variable is defined, and 10 to the power of that variable value is sent back to the generator. 100 is sent back, and since it is different from None
, num is equated to i and becomes 100, then it’s summed with 1 and becomes 101, the next palindrome. Here is what I don’t get: when that number is throw and correctly identified as a palindrome, the code goes back to the for e in pal_gen:
line, but it doesn’t enter the block. Only the next palindrome, 111
is sent back down and enters the block, where the digits
variable is defined again, and so on and so forth, and this patterns repeats, with only every other palindrome entering the block. Why is that?
Can someone, please, explain to me why that happens?
To understand the behaviour, we need to understand how for
loops are implemented in Python. This code:
for e in pal_gen:
digits = len(str(e))
pal_gen.send(10**(digits))
is basically equivalent to (except that no name _iter
is actually generated):
_iter = iter(pal_gen)
while True:
try:
e = next(_iter)
digits = len(str(e))
pal_gen.send(10**(digits))
except StopIteration:
break
However, because pal_gen
is a generator object (created by calling the generator function infinite_palindromes
), it is already an iterator; calling iter
on it returns the same object unchanged. Thus, we effectively have:
while True:
try:
e = next(pal_gen)
digits = len(str(e))
pal_gen.send(10**(digits))
except StopIteration:
break
The other thing to note here is that calling .send
on a generator advances the generator, and so does passing it to next
(i.e., using it like any other iterator). It’s just that the code was written to ignore the value retrieved using .send
.
In fact, next
on a generator is equivalent to calling .send
and passing None
(by convention; Python defines the behaviour this way so that the assignment in the generator code, i = (yield num)
, has something to assign).
So, each time through the loop, two values are sent to the generator:
while True:
try:
e = pal_gen.send(None)
digits = len(str(e))
pal_gen.send(10**(digits))
except StopIteration:
break
The control flow is like so:
None
is sent to the generator. Because it hasn’t yielded yet, this value is discarded (Python requires it to beNone
, since it can’t be used by the generator).- The generator iterates its own loop until
11
is found and yielded. The execution of the generator pauses here:i
is not assigned yet. - The main loop receives
11
as a value fore
, computes100
as the next value to work with, and sends that to the generator. - The generator receives
100
and assigns it toi
, then proceeds with its logic. The next step isif i is not None:
, and that is indeed the case – thus,100
is assigned tonum
.num
is incremented at the end of the loop. - The loop inside the generator runs into the next iteration, and immediately finds that
101
is a palindrome. Thus, the.send
call returns a value of101
, but this isn’t assigned anywhere. - Similarly, in the second iteration of the main loop,
111
will be found and yielded (as the next palindrome after101
) and assigned toe
in the first call; then1000
is sent explicitly, causing1001
to be yielded and ignored. In the third iteration,1111
is assigned toe
, then10001
is ignored. In the fourth iteration,10101
(not11111
! This is the next palindrome after10001
) is assigned, and100001
is ignored. Etc.
If the code is tested at the REPL, the values returned from the explicit .send
will be displayed. This is just a quirk of the REPL.