Is Python *with* statement exactly equivalent to a try – (except) – finally block?

Question:

I know this was widely discussed, but I still can’t find an answer to confirm this: is the with statement identical to calling the same code in a try – (except) -finally block, where whatever one defines in the __exit__ function of the context manager is placed in the finally block?

For example — are these 2 code snippets doing exactly the same thing?

import sys
from contextlib import contextmanager

@contextmanager
def open_input(fpath):
    fd = open(fpath) if fpath else sys.stdin
    try:
        yield fd
    finally:
        fd.close()

with open_input("/path/to/file"):
    print "starting to read from file..."

the same as:

def open_input(fpath):
    try:
        fd = open(fpath) if fpath else sys.stdin
        print "starting to read from file..."
    finally:
        fd.close()

open_input("/path/to/file")

Thanks!

Asked By: Clara

||
Source

Answers:

I’m going to put aside mentions of scope, because it’s really not very relevant.

According to PEP 343,

with EXPR as VAR:
    BLOCK

translates to

mgr = (EXPR)
exit = type(mgr).__exit__  # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
    try:
        VAR = value  # Only if "as VAR" is present
        BLOCK
    except:
        # The exceptional case is handled here
        exc = False
        if not exit(mgr, *sys.exc_info()):
            raise
        # The exception is swallowed if exit() returns true
finally:
    # The normal and non-local-goto cases are handled here
    if exc:
        exit(mgr, None, None, None)

As you can see, type(mgr).__enter__ is called as you expect, but not inside the try.

type(mgr).__exit__ is called on exit. The only difference is that when there is an exception, the if not exit(mgr, *sys.exc_info()) path is taken. This gives with the ability to introspect and silence errors unlike what a finally clause can do.

contextmanager doesn’t complicate this much. It’s just:

def contextmanager(func):
    @wraps(func)
    def helper(*args, **kwds):
        return _GeneratorContextManager(func, *args, **kwds)
    return helper

Then look at the class in question:

class _GeneratorContextManager(ContextDecorator):
    def __init__(self, func, *args, **kwds):
        self.gen = func(*args, **kwds)

    def __enter__(self):
        try:
            return next(self.gen)
        except StopIteration:
            raise RuntimeError("generator didn't yield") from None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
                next(self.gen)
            except StopIteration:
                return
            else:
                raise RuntimeError("generator didn't stop")
        else:
            if value is None:
                value = type()
            try:
                self.gen.throw(type, value, traceback)
                raise RuntimeError("generator didn't stop after throw()")
            except StopIteration as exc:
                return exc is not value
            except:
                if sys.exc_info()[1] is not value:
                    raise

Unimportant code has been elided.

The first thing to note is that if there are multiple yields, this code will error.

This does not affect the control flow noticeably.

Consider __enter__.

try:
    return next(self.gen)
except StopIteration:
    raise RuntimeError("generator didn't yield") from None

If the context manager was well written, this will never break from what is expected.

One difference is that if the generator throws StopIteration, a different error (RuntimeError) will be produced. This means the behaviour is not totally identical to a normal with if you’re running completely arbitrary code.

Consider a non-erroring __exit__:

if type is None:
    try:
        next(self.gen)
    except StopIteration:
        return
    else:
        raise RuntimeError("generator didn't stop")

The only difference is as before; if your code throws StopIteration, it will affect the generator and thus the contextmanager decorator will misinterpret it.

This means that:

from contextlib import contextmanager

@contextmanager
def with_cleanup(func):
    try:
        yield
    finally:
        func()

def good_cleanup():
    print("cleaning")

with with_cleanup(good_cleanup):
    print("doing")
    1/0
#>>> doing
#>>> cleaning
#>>> Traceback (most recent call last):
#>>>   File "", line 15, in <module>
#>>> ZeroDivisionError: division by zero

def bad_cleanup():
    print("cleaning")
    raise StopIteration

with with_cleanup(bad_cleanup):
    print("doing")
    1/0
#>>> doing
#>>> cleaning

Which is unlikely to matter, but it could.

Finally:

else:
    if value is None:
        value = type()
    try:
        self.gen.throw(type, value, traceback)
        raise RuntimeError("generator didn't stop after throw()")
    except StopIteration as exc:
        return exc is not value
    except:
        if sys.exc_info()[1] is not value:
            raise

This raises the same question about StopIteration, but it’s interesting to note that last part.

if sys.exc_info()[1] is not value:
    raise

This means that if the exception is unhandled, the traceback will be unchanged. If it was handled but a new traceback exists, that will be raised instead.

This perfectly matches the spec.

TL;DR

  • with is actually slightly more powerful than a try...finally in that the with can introspect and silence errors.

  • Be careful about StopIteration, but otherwise you’re fine using @contextmanager to create context managers.

Answered By: Veedrac