Understanding the Python with statement and context managers

The with in Python is intended for wrapping a set of statements where you should set up and destroy or close resources. It is in a way similar to try...finally in that regard as the finally clause will be executed even after an exception.

A context manager is an object that implements two methods: __enter__ and __exit__. Those are called immediately before and after (respectively) the with block.

For instance, take a look at the classic open() example:

with open('temp.txt', 'w') as f:
    f.write("Hi!")

Open returns a File object that implements __enter__ more or less like return self and __exit__ like self.close().

The contextlib.contextmanager function decorator provides a handy way of providing a context manager without the need to write a full-fledged ContextManager class of your own (with __enter__ and __exit__ methods, so you don’t have to remember the arguments to the __exit__ method, or that the __exit__ method must return True in order to suppress the exception). Instead, you write a function with a single yield at the point you want the with block to run, and you trap any exceptions (that effectively come from the yield) as you normally would.

from contextlib import contextmanager
@contextmanager
def handler():
    # Put here what would ordinarily go in the `__enter__` method
    # In this case, there's nothing to do
    try:
        yield # You can return something if you want, that gets picked up in the 'as'
    except Exception as e:
        print "no not possible"
    finally:
        print "Ok I caught you"

with handler():
    name='rubicon'/2 #to raise an exception

Why go to the extra trouble of writing a context manager? Code re-use. You can use the same context manager in multiple places, without having to duplicate the exception handling. If the exception handling is unique to that situation, then don’t bother with a context manager. But if the same pattern crops up again and again (or if it might for your users, e.g., closing a file, unlocking a mutex), it’s worth the extra trouble. It’s also a neat pattern to use if the exception handling is a bit complicated, as it separates the exception handling from the main line of code flow.

with statements or context managers are there to aid with resources (although may be used for much more).

Let’s say you opened a file for writing:

f = open(path, "w")

You now have an open file handle. During the handling of your file, no other program can write to it. In order to let other programs write to it, you must close the file handle:

f.close()

But, before closing your file an error occured:

f = open(path, "w")
data = 3/0  # Tried dividing by zero. Raised ZeroDivisionError
f.write(data)
f.close()

What will happen now is that the function or entire program will exit, while leaving your file with an open handle. (CPython cleans handles on termination and handles are freed together with a program but you shouldn’t count on that)

A with statement ensures that as soon as you leave it’s indentation, it will close the file handle:

with open(path, "w") as f:
    data = 3/0  # Tried dividing by zero. Raised ZeroDivisionError
    f.write(data)
# In here the file is already closed automatically, no matter what happened.

---

with statements may be used for many more things. For example: threading.Lock()

lock = threading.Lock()
with lock:  # Lock is acquired
   do stuff...
# Lock is automatically released.

Almost everything done with a context manager can be done with try: ... finally: ... but context managers are nicer to use, more comfortable, more readable and by implementing __enter__ and __exit__ provide an easy to use interface.

---

Creating context managers is done by implementing __enter__() and __exit__() in a normal class.

__enter__() tells what to do when a context manager starts and __exit__() when a context manager exists (giving the exception to the __exit__() method if an exception occurred)

A shortcut for creating context managers can be found in contextlib. It wraps a generator as a context manager.

The Components of Context Manager

1. You should implement an __enter__ method that returns an object
2. Implement a __exit__ method.

Example

I will give a simple example to show you why we need a context manager. During the winter of Xinjiang, China, you should close a door immediately when you open a door. if you forget to close it, you will get cold.

 class Door:
     def __init__(self):
         self.doorstatus='the door was closed when you are not at home'
         print(self.doorstatus)
     def __enter__(self):
         print('I have opened the door')
         return self
     def __exit__(self,*args):
         print('pong!the door has closed')
     def fetchsomethings(self):
         print('I have fetched somethings')

when fetch things at home, you should open a door, fetch somethings and close the door.

 with Door() as dr:
     dr.fetchsomethings()

the output is:

the door was closed when you are not at home
I have opened the door
I have fetched somethings
pong!the door has closed

Explanation

when you initiate a Door class, it will call __init__ method that will print
“the door was closed when you are not in home” and __enter__ method that will print “I have opened the door” and return a door instance called dr. when call self.fetchsomethings in with block, the method will print “I have fetched somethings”.when the block is finished.the context manager will invoke __exit__
method and it will print “pong!the door has closed” .when you do not use with
keyword ,__enter__and __exit__ will not be invoked!!!!

Managing Resources: In any programming language, the usage of resources like file operations or database connections is very common. But these resources are limited in supply. Therefore, the main problem lies in making sure to release these resources after usage. If they are not released then it will lead to resource leakage and may cause the system to either slow down or crash. It would be very helpful if users have a mechanism for the automatic setup and teardown of resources. In Python, it can be achieved by the usage of context managers which facilitate the proper handling of resources. The most common way of performing file operations is by using the keyword as shown below:

 # Python program showing a use of with keyword
 with open("test.txt") as f:  
     data = f.read()

When a file is opened, a file descriptor is consumed which is a limited resource. Only a certain number of files can be opened by a process at a time. The following program demonstrates it.

file_descriptors = []
for x in range(100000):
    file_descriptors.append(open('test.txt', 'w'))

it lead the error: OSError: [Errno 24] Too many open files: 'test.txt'

Python provides an easy way to manage resources: Context Managers. The with keyword is used. When it gets evaluated it should result in an object that performs context management. Context managers can be written using classes or functions(with decorators).

Creating a Context Manager: When creating context managers using classes, user need to ensure that the class has the methods: __enter__() and __exit__(). The __enter__() returns the resource that needs to be managed and the __exit__() does not return anything but performs the cleanup operations. First, let us create a simple class called ContextManager to understand the basic structure of creating context managers using classes, as shown below:

# Python program creating a context manager 
class ContextManager():
    def __init__(self):
        print('init method called')
         
    def __enter__(self):
        print('enter method called')
        return self
     
    def __exit__(self, exc_type, exc_value, exc_traceback):
        print('exit method called')
 
with ContextManager() as manager:
    print('with statement block')

Output:
init method called
enter method called
with statement block
exit method called

In this case, a ContextManager object is created. This is assigned to the variable after the keyword i.e manager. On running the above program, the following get executed in sequence:

• __init__()
• __enter__()
• statement body (code inside the with block)
• __exit__()[the parameters in this method are used to manage exceptions]