How are generators and coroutines implemented in CPython?
Question:
I’ve read that in CPython, the interpreter stack (the list of Python functions called to reach this point) is mixed with the C stack (the list of C functions that were called in the interpreter’s own code). If so, then how are generators and coroutines implemented? How do they remember their execution state? Does CPython copy each generator’s / coroutine’s stack to and from an OS stack? Or does CPython simply keep the generator’s topmost stack frame on the heap, since the generator can only yield from that topmost frame?
Answers:
The yield instruction takes the current executing context as a closure, and transforms it into an own living object. This object has a __iter__ method which will continue after this yield statement.
So the call stack gets transformed into a heap object.
The notion that Python’s stack and C stack in a running Python program are intermixed can be misleading.
The Python stack is something completely separated than the actual C stack used by the interpreter. The data structures on Python stack are actually full Python “frame” objects (that can even be introspected and have some attributes changed at run time). This stack is managed by the Python virtual machine, which itself runs in C and thus have a normal C program, machine level, stack.
When using generators and iterators, the interpreter simply stores the respective frame object somewhere else than on the Python program stack, and pushes it back there when execution of the generator resumes. This “somewhere else” is the generator object itself.Calling the method “next” or “send” on the generator object causes this to happen.
I’ve read that in CPython, the interpreter stack (the list of Python functions called to reach this point) is mixed with the C stack (the list of C functions that were called in the interpreter’s own code). If so, then how are generators and coroutines implemented? How do they remember their execution state? Does CPython copy each generator’s / coroutine’s stack to and from an OS stack? Or does CPython simply keep the generator’s topmost stack frame on the heap, since the generator can only yield from that topmost frame?
The yield instruction takes the current executing context as a closure, and transforms it into an own living object. This object has a __iter__ method which will continue after this yield statement.
So the call stack gets transformed into a heap object.
The notion that Python’s stack and C stack in a running Python program are intermixed can be misleading.
The Python stack is something completely separated than the actual C stack used by the interpreter. The data structures on Python stack are actually full Python “frame” objects (that can even be introspected and have some attributes changed at run time). This stack is managed by the Python virtual machine, which itself runs in C and thus have a normal C program, machine level, stack.
When using generators and iterators, the interpreter simply stores the respective frame object somewhere else than on the Python program stack, and pushes it back there when execution of the generator resumes. This “somewhere else” is the generator object itself.Calling the method “next” or “send” on the generator object causes this to happen.