Pythonic way to compose context managers for objects owned by a class

Question:

It’s typical to require for some task multiple objects which have resources to be explicitly released – say, two files; this is easily done when the task is local to a function using nested with blocks, or – even better – a single with block with multiple with_item clauses:

with open('in.txt', 'r') as i, open('out.txt', 'w') as o:
    # do stuff

OTOH, I still struggle to understand how this is supposed to work when such objects aren’t just local to a function scope, but owned by a class instance – in other words, how context managers compose.

Ideally I’d like to do something like:

class Foo:
    def __init__(self, in_file_name, out_file_name):
        self.i = WITH(open(in_file_name, 'r'))
        self.o = WITH(open(out_file_name, 'w'))

and have Foo itself turn into a context manager that handles i and o, such that when I do

with Foo('in.txt', 'out.txt') as f:
    # do stuff

self.i and self.o are taken care of automatically as you would expect.

I tinkered about writing stuff such as:

class Foo:
    def __init__(self, in_file_name, out_file_name):
        self.i = open(in_file_name, 'r').__enter__()
        self.o = open(out_file_name, 'w').__enter__()

    def __enter__(self):
        return self

    def __exit__(self, *exc):
        self.i.__exit__(*exc)
        self.o.__exit__(*exc)

but it’s both verbose and unsafe against exceptions occurring in the constructor. After searching for a while, I found this 2015 blog post, which uses contextlib.ExitStack to obtain something very similar to what I’m after:

class Foo(contextlib.ExitStack):
    def __init__(self, in_file_name, out_file_name):
        super().__init__()
        self.in_file_name = in_file_name
        self.out_file_name = out_file_name

    def __enter__(self):
        super().__enter__()
        self.i = self.enter_context(open(self.in_file_name, 'r')
        self.o = self.enter_context(open(self.out_file_name, 'w')
        return self

This is pretty satisfying, but I’m perplexed by the fact that:

  • I find nothing about this usage in the documentation, so it doesn’t seem to be the “official” way to tackle this problem;
  • in general, I find it extremely difficult to find information about this issue, which makes me think I’m trying to apply an unpythonic solution to the problem.

Some extra context: I work mostly in C++, where there is no distinction between the block-scope case and the object-scope case for this issue, as this kind of cleanup is implemented inside the destructor (think __del__, but invoked deterministically), and the destructor (even if not explicitly defined) automatically invokes the destructors of the subobjects. So both:

{
    std::ifstream i("in.txt");
    std::ofstream o("out.txt");
    // do stuff
}

and

struct Foo {
    std::ifstream i;
    std::ofstream o;

    Foo(const char *in_file_name, const char *out_file_name) 
        : i(in_file_name), o(out_file_name) {}
}

{
    Foo f("in.txt", "out.txt");
}

do all the cleanup automatically as you generally want.

I’m looking for a similar behavior in Python, but again, I’m afraid I’m just trying to apply a pattern coming from C++, and that the underlying problem has a radically different solution that I can’t think of.


So, to sum it up: what is the Pythonic solution to the problem of having an object who owns objects that require cleanup become a context-manager itself, calling correctly the __enter__/__exit__ of its children?

Asked By: Matteo Italia

||

Answers:

I think use a helper is better:

from contextlib import ExitStack, contextmanager

class Foo:
    def __init__(self, i, o):
        self.i = i
        self.o = o

@contextmanager
def multiopen(i, o):
    with ExitStack() as stack:
        i = stack.enter_context(open(i))
        o = stack.enter_context(open(o))
        yield Foo(i, o)

The usage is close to native open:

with multiopen(i_name, o_name) as foo:
    pass
Answered By: Sraw

Well, if you want to process certainly for file handlers, the simplest solution is just to pass filehandlers directly to your class instead of filenames.

with open(f1, 'r') as f1, open(f2, 'w') as f2:
   with MyClass(f1, f2) as my_obj:
       ...

If you don’t need custom __exit__ functionality you may even skip nested with.

If you really want to pass filenames to __init__, your problem could be solved like this:

class MyClass:
     input, output = None, None

     def __init__(self, input, output):
         try:
             self.input = open(input, 'r')
             self.output = open(output, 'w')
         except BaseException as exc:
             self.__exit___(type(exc), exc, exc.__traceback__)
             raise

     def __enter__(self):
         return self

     def __exit__(self, *args):
            self.input and self.input.close()
            self.output and self.output.close()
        # My custom __exit__ code

So, it really depends on your task, python have plenty options to work with. In the end of the day – pythonic way is to keep your api simple.

Answered By: Victor Gavro

I think contextlib.ExitStack is Pythonic and canonical and it’s the appropriate solution to this problem. The rest of this answer tries to show the links I used to come to this conclusion and my thought process:

Original Python enhancement request

https://bugs.python.org/issue13585

The original idea + implementation was proposed as a Python standard library enhancement with both reasoning and sample code. It was discussed in detail by such core developers as Raymond Hettinger and Eric Snow. The discussion on this issue clearly shows the growth of the original idea into something that is applicable for the standard library and is Pythonic. Attempted summarization of the thread is:

nikratio originally proposed:

I’d like to propose addding the CleanupManager class described in http://article.gmane.org/gmane.comp.python.ideas/12447 to the contextlib module. The idea is to add a general-purpose context manager to manage (python or non-python) resources that don’t come with their own context manager

Which was met with concerns from rhettinger:

So far, there has been zero demand for this and I’ve not seen code like it being used in the wild. AFAICT, it is not demonstrably better than a straight-forward try/finally.

As a response to this there was a long discussion about whether there was a need for this, leading to posts like these from ncoghlan:

TestCase.setUp() and TestCase.tearDown() were amongst the precursors to__enter__() and exit(). addCleanUp() fills exactly the same role here – and I’ve seen plenty of positive feedback directed towards Michael for that addition to the unittest API…
…Custom context managers are typically a bad idea in these circumstances, because they make readability worse (relying on people to understand what the context manager does). A standard library based solution, on the other hand, offers the best of both worlds:
– code becomes easier to write correctly and to audit for correctness (for all the reasons with statements were added in the first place)
– the idiom will eventually become familiar to all Python users…
…I can take this up on python-dev if you want, but I hope to persuade you that the desire is there…

And then again from ncoghlan a little later:

My earlier descriptions here aren’t really adequate – as soon as I started putting contextlib2 together, this CleanupManager idea quickly morphed into ContextStack [1], which is a far more powerful tool for manipulating context managers in a way that doesn’t necessarily correspond with lexical scoping in the source code.

Examples / recipes / blog posts of ExitStack
There are several examples and recipes within the standard library source code itself, which you can see in the merge revision that added this feature: https://hg.python.org/cpython/rev/8ef66c73b1e1

There is also a blog post from the original issue creator (Nikolaus Rath / nikratio) that describes in a compelling way why ContextStack is a good pattern and also provides some usage examples: https://www.rath.org/on-the-beauty-of-pythons-exitstack.html

Answered By: Matthew Horst

Your second example is the most straight forward way to do it in
Python (i.e., most Pythonic). However, your example still has a bug. If
an exception is raised during
the second open(),

self.i = self.enter_context(open(self.in_file_name, 'r')
self.o = self.enter_context(open(self.out_file_name, 'w') # <<< HERE

then self.i will not be released when you expect because
Foo.__exit__() will not be called unless Foo.__enter__() successfully
returns. To fix this, wrap each context call in a
try-except that will call Foo.__exit__() when an exception occurs.

import contextlib
import sys

class Foo(contextlib.ExitStack):

    def __init__(self, in_file_name, out_file_name):
        super().__init__()
        self.in_file_name = in_file_name
        self.out_file_name = out_file_name

    def __enter__(self):
        super().__enter__()

        try:
            # Initialize sub-context objects that could raise exceptions here.
            self.i = self.enter_context(open(self.in_file_name, 'r'))
            self.o = self.enter_context(open(self.out_file_name, 'w'))

        except:
            if not self.__exit__(*sys.exc_info()):
                raise

        return self
Answered By: Uyghur Lives Matter

As @cpburnz mentioned, your last example is best, but does contain a bug if the second open fails. Avoiding this bug is described in the standard library documentation. We can easily adapt the code snippets from the ExitStack documentation and the example for ResourceManager from 29.6.2.4 Cleaning up in an __enter__ implementation to come up with a MultiResourceManager class:

from contextlib import contextmanager, ExitStack
class MultiResourceManager(ExitStack):
    def __init__(self, resources, acquire_resource, release_resource,
            check_resource_ok=None):
        super().__init__()
        self.acquire_resource = acquire_resource
        self.release_resource = release_resource
        if check_resource_ok is None:
            def check_resource_ok(resource):
                return True
        self.check_resource_ok = check_resource_ok
        self.resources = resources
        self.wrappers = []

    @contextmanager
    def _cleanup_on_error(self):
        with ExitStack() as stack:
            stack.push(self)
            yield
            # The validation check passed and didn't raise an exception
            # Accordingly, we want to keep the resource, and pass it
            # back to our caller
            stack.pop_all()

    def enter_context(self, resource):
        wrapped = super().enter_context(self.acquire_resource(resource))
        if not self.check_resource_ok(wrapped):
            msg = "Failed validation for {!r}"
            raise RuntimeError(msg.format(resource))
        return wrapped

    def __enter__(self):
        with self._cleanup_on_error():
            self.wrappers = [self.enter_context(r) for r in self.resources]
        return self.wrappers

    # NB: ExitStack.__exit__ is already correct

Now your Foo() class is trivial:

import io
class Foo(MultiResourceManager):
    def __init__(self, *paths):
        super().__init__(paths, io.FileIO, io.FileIO.close)

This is nice because we don’t need any try-except blocks — you’re probably only using ContextManagers to get rid of those in the first place!

Then you can use it like you wanted to (note MultiResourceManager.__enter__ returns a list of objects given by the passed acquire_resource()):

if __name__ == '__main__':
    open('/tmp/a', 'w').close()
    open('/tmp/b', 'w').close()

    with Foo('/tmp/a', '/tmp/b') as (f1, f2):
        print('opened {0} and {1}'.format(f1.name, f2.name))

We can replace io.FileIO with debug_file as in the following snippet to see it in action:

    class debug_file(io.FileIO):
        def __enter__(self):
            print('{0}: enter'.format(self.name))
            return super().__enter__()
        def __exit__(self, *exc_info):
            print('{0}: exit'.format(self.name))
            return super().__exit__(*exc_info)

Then we see:

/tmp/a: enter
/tmp/b: enter
opened /tmp/a and /tmp/b
/tmp/b: exit
/tmp/a: exit

If we added import os; os.unlink('/tmp/b') just before the loop we’d see:

/tmp/a: enter
/tmp/a: exit
Traceback (most recent call last):
  File "t.py", line 58, in <module>
    with Foo('/tmp/a', '/tmp/b') as (f1, f2):
  File "t.py", line 46, in __enter__
    self.wrappers = [self.enter_context(r) for r in self.resources]
  File "t.py", line 46, in <listcomp>
    self.wrappers = [self.enter_context(r) for r in self.resources]
  File "t.py", line 38, in enter_context
    wrapped = super().enter_context(self.acquire_resource(resource))
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/b'

You can see /tmp/a is closed correctly.

Answered By: fritzelr
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.