Handling exceptions inside context managers
Question:
I have some code where I try to reach a resource but sometimes it is unavailable, and results in an exception. I tried to implement a retry engine using context managers, but I can’t handle the exception raised by the caller inside the __enter__ context for my context manager.
class retry(object):
def __init__(self, retries=0):
self.retries = retries
self.attempts = 0
def __enter__(self):
for _ in range(self.retries):
try:
self.attempts += 1
return self
except Exception as e:
err = e
def __exit__(self, exc_type, exc_val, traceback):
print 'Attempts', self.attempts
These are some examples which just raise an exception (which I expected to handle)
>>> with retry(retries=3):
... print ok
...
Attempts 1
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
NameError: name 'ok' is not defined
>>>
>>> with retry(retries=3):
... open('/file')
...
Attempts 1
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
IOError: [Errno 2] No such file or directory: '/file'
Is there any way to intercept these exception(s) and handle them inside the context manager?
Answers:
To deal with an exception in an __enter__ method, the most straightforward (and less surprising) thing to do, would be to wrap the with statement itself in a try-except clause, and simply raise the exception –
But, with blocks are definetelly not designed to work like this – to be, by themselves “retriable” – and there is some misunderstanding here:
def __enter__(self):
for _ in range(self.retries):
try:
self.attempts += 1
return self
except Exception as e:
err = e
Once you return self there, the context were __enter__ runs no longer exists – if an error occurs inside the with block, it will just flow naturally to the __exit__ method. And no, the __exit__ method can not, in anyway, make the execution flow go back to the beginning of the with block.
You are probably wanting something more like this:
class Retrier(object):
max_retries = 3
def __init__(self, ...):
self.retries = 0
self.acomplished = False
def __enter__(self):
return self
def __exit__(self, exc, value, traceback):
if not exc:
self.acomplished = True
return True
self.retries += 1
if self.retries >= self.max_retries:
return False
return True
....
x = Retrier()
while not x.acomplished:
with x:
...
Quoting __exit__,
If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.
By default, if you don’t return a value explicitly from a function, Python will return None, which is a falsy value. In your case, __exit__ returns None and that is why the exeception is allowed to flow past the __exit__.
So, return a truthy value, like this
class retry(object):
def __init__(self, retries=0):
...
def __enter__(self):
...
def __exit__(self, exc_type, exc_val, traceback):
print 'Attempts', self.attempts
print exc_type, exc_val
return True # or any truthy value
with retry(retries=3):
print ok
the output will be
Attempts 1
<type 'exceptions.NameError'> name 'ok' is not defined
If you want to have the retry functionality, you can implement that with a decorator, like this
def retry(retries=3):
left = {'retries': retries}
def decorator(f):
def inner(*args, **kwargs):
while left['retries']:
try:
return f(*args, **kwargs)
except NameError as e:
print e
left['retries'] -= 1
print "Retries Left", left['retries']
raise Exception("Retried {} times".format(retries))
return inner
return decorator
@retry(retries=3)
def func():
print ok
func()
I think this one is easy, and other folks seem to be overthinking it. Just put the resource fetching code in __enter__, and try to return, not self, but the resource fetched. In code:
def __init__(self, retries):
...
# for demo, let's add a list to store the exceptions caught as well
self.errors = []
def __enter__(self):
for _ in range(self.retries):
try:
return resource # replace this with real code
except Exception as e:
self.attempts += 1
self.errors.append(e)
# this needs to return True to suppress propagation, as others have said
def __exit__(self, exc_type, exc_val, traceback):
print 'Attempts', self.attempts
for e in self.errors:
print e # as demo, print them out for good measure!
return True
Now try it:
>>> with retry(retries=3) as resource:
... # if resource is successfully fetched, you can access it as `resource`;
... # if fetching failed, `resource` will be None
... print 'I get', resource
I get None
Attempts 3
name 'resource' is not defined
name 'resource' is not defined
name 'resource' is not defined
You don’t have to implement the retry functionality manually. Take a look at the tenacity library.
Tenacity is a general-purpose retrying library, written in Python, to
simplify the task of adding retry behavior to just about anything.
You can simply add @retry decorator with parameters to your function.
Also,
Tenacity allows you to retry a code block without the need to wraps it
in an isolated function. The trick is to combine a for loop and a
context manager.
I found contextmanager from contextlib useful, hope this may be helpful.
from contextlib import contextmanager
@contextmanager
def handler(*args, **kwargs):
try:
# print(*args, **kwargs)
yield
except Exception:
# Handle exception
Now, to use it,
# Add optional args or kwargs
with handler():
# Code with probable exception
print("Hi")
I have some code where I try to reach a resource but sometimes it is unavailable, and results in an exception. I tried to implement a retry engine using context managers, but I can’t handle the exception raised by the caller inside the __enter__ context for my context manager.
class retry(object):
def __init__(self, retries=0):
self.retries = retries
self.attempts = 0
def __enter__(self):
for _ in range(self.retries):
try:
self.attempts += 1
return self
except Exception as e:
err = e
def __exit__(self, exc_type, exc_val, traceback):
print 'Attempts', self.attempts
These are some examples which just raise an exception (which I expected to handle)
>>> with retry(retries=3):
... print ok
...
Attempts 1
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
NameError: name 'ok' is not defined
>>>
>>> with retry(retries=3):
... open('/file')
...
Attempts 1
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
IOError: [Errno 2] No such file or directory: '/file'
Is there any way to intercept these exception(s) and handle them inside the context manager?
To deal with an exception in an __enter__ method, the most straightforward (and less surprising) thing to do, would be to wrap the with statement itself in a try-except clause, and simply raise the exception –
But, with blocks are definetelly not designed to work like this – to be, by themselves “retriable” – and there is some misunderstanding here:
def __enter__(self):
for _ in range(self.retries):
try:
self.attempts += 1
return self
except Exception as e:
err = e
Once you return self there, the context were __enter__ runs no longer exists – if an error occurs inside the with block, it will just flow naturally to the __exit__ method. And no, the __exit__ method can not, in anyway, make the execution flow go back to the beginning of the with block.
You are probably wanting something more like this:
class Retrier(object):
max_retries = 3
def __init__(self, ...):
self.retries = 0
self.acomplished = False
def __enter__(self):
return self
def __exit__(self, exc, value, traceback):
if not exc:
self.acomplished = True
return True
self.retries += 1
if self.retries >= self.max_retries:
return False
return True
....
x = Retrier()
while not x.acomplished:
with x:
...
Quoting __exit__,
If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.
By default, if you don’t return a value explicitly from a function, Python will return None, which is a falsy value. In your case, __exit__ returns None and that is why the exeception is allowed to flow past the __exit__.
So, return a truthy value, like this
class retry(object):
def __init__(self, retries=0):
...
def __enter__(self):
...
def __exit__(self, exc_type, exc_val, traceback):
print 'Attempts', self.attempts
print exc_type, exc_val
return True # or any truthy value
with retry(retries=3):
print ok
the output will be
Attempts 1
<type 'exceptions.NameError'> name 'ok' is not defined
If you want to have the retry functionality, you can implement that with a decorator, like this
def retry(retries=3):
left = {'retries': retries}
def decorator(f):
def inner(*args, **kwargs):
while left['retries']:
try:
return f(*args, **kwargs)
except NameError as e:
print e
left['retries'] -= 1
print "Retries Left", left['retries']
raise Exception("Retried {} times".format(retries))
return inner
return decorator
@retry(retries=3)
def func():
print ok
func()
I think this one is easy, and other folks seem to be overthinking it. Just put the resource fetching code in __enter__, and try to return, not self, but the resource fetched. In code:
def __init__(self, retries):
...
# for demo, let's add a list to store the exceptions caught as well
self.errors = []
def __enter__(self):
for _ in range(self.retries):
try:
return resource # replace this with real code
except Exception as e:
self.attempts += 1
self.errors.append(e)
# this needs to return True to suppress propagation, as others have said
def __exit__(self, exc_type, exc_val, traceback):
print 'Attempts', self.attempts
for e in self.errors:
print e # as demo, print them out for good measure!
return True
Now try it:
>>> with retry(retries=3) as resource:
... # if resource is successfully fetched, you can access it as `resource`;
... # if fetching failed, `resource` will be None
... print 'I get', resource
I get None
Attempts 3
name 'resource' is not defined
name 'resource' is not defined
name 'resource' is not defined
You don’t have to implement the retry functionality manually. Take a look at the tenacity library.
Tenacity is a general-purpose retrying library, written in Python, to
simplify the task of adding retry behavior to just about anything.
You can simply add @retry decorator with parameters to your function.
Also,
Tenacity allows you to retry a code block without the need to wraps it
in an isolated function. The trick is to combine a for loop and a
context manager.
I found contextmanager from contextlib useful, hope this may be helpful.
from contextlib import contextmanager
@contextmanager
def handler(*args, **kwargs):
try:
# print(*args, **kwargs)
yield
except Exception:
# Handle exception
Now, to use it,
# Add optional args or kwargs
with handler():
# Code with probable exception
print("Hi")