Jupyter magic to handle notebook exceptions
Question:
I have a few long-running experiments in my Jupyter Notebooks. Because I don’t know when they will finish, I add an email function to the last cell of the notebook, so I automatically get an email, when the notebook is done.
But when there is a random exception in one of the cells, the whole notebook stops executing and I never get any email. So I’m wondering if there is some magic function that could execute a function in case of an exception / kernel stop.
Like
def handle_exception(stacktrace):
send_mail_to_myself(stacktrace)
%%in_case_of_notebook_exception handle_exception # <--- this is what I'm looking for
The other option would be to encapsulate every cell in try-catch, right? But that’s soooo tedious.
Thanks in advance for any suggestions.
Answers:
I don’t think there is an out-of-the-box way to do that not using a try..except statement in your cells. AFAIK a 4 years old issue mentions this, but is still in open status.
However, the runtools extension may do the trick.
Such a magic command does not exist, but you can write it yourself.
from IPython.core.magic import register_cell_magic
@register_cell_magic('handle')
def handle(line, cell):
try:
exec(cell)
except Exception as e:
send_mail_to_myself(e)
raise # if you want the full trace-back in the notebook
It is not possible to load the magic command for the entire notebook automatically, you have to add it at each cell where you need this feature.
%%handle
some_code()
raise ValueError('this exception will be caught by the magic command')
@show0k gave the correct answer to my question (in regards to magic methods). Thanks a lot! 🙂
That answer inspired me to dig a little deeper and I came across an IPython method that lets you define a custom exception handler for the whole notebook.
I got it to work like this:
from IPython.core.ultratb import AutoFormattedTB
# initialize the formatter for making the tracebacks into strings
itb = AutoFormattedTB(mode = 'Plain', tb_offset = 1)
# this function will be called on exceptions in any cell
def custom_exc(shell, etype, evalue, tb, tb_offset=None):
# still show the error within the notebook, don't just swallow it
shell.showtraceback((etype, evalue, tb), tb_offset=tb_offset)
# grab the traceback and make it into a list of strings
stb = itb.structured_traceback(etype, evalue, tb)
sstb = itb.stb2text(stb)
print (sstb) # <--- this is the variable with the traceback string
print ("sending mail")
send_mail_to_myself(sstb)
# this registers a custom exception handler for the whole current notebook
get_ipython().set_custom_exc((Exception,), custom_exc)
So this can be put into a single cell at the top of any notebook and as a result it will do the mailing in case something goes wrong.
Note to self / TODO: make this snippet into a little python module that can be imported into a notebook and activated via line magic.
Be careful though. The documentation contains a warning for this set_custom_exc method: “WARNING: by putting in your own exception handler into IPython’s main execution loop, you run a very good chance of nasty crashes. This facility should only be used if you really know what you are doing.”
Why exec is not always the solution
It’s some years later and I had a similar issue trying to handle errors with Jupyter magics. However, I needed variables to persist in the actual Jupyter notebook.
%%try_except print
a = 12
raise ValueError('test')
In this example, I want the error to print (but could be anything such as e-mail as in the opening post), but also a == 12 to be true in the next cell. For that reason, the method exec suggested does not work when you define the magic in a different file. The solution I found is to use the IPython functionalities.
How you can solve it
from IPython.core.magic import line_magic, cell_magic, line_cell_magic, Magics, magics_class
@magics_class
class CustomMagics(Magics):
@cell_magic
def try_except(self, line, cell):
""" This magic wraps a cell in try_except functionality """
try:
self.shell.ex(cell) # This executes the cell in the current namespace
except Exception as e:
if ip.ev(f'callable({how})'): # check we have a callable handler
self.shell.user_ns['error'] = e # add error to namespace
ip.ev(f'{how}(error)') # call the handler with the error
else:
raise e
# Register
from IPython import get_ipython
ip = get_ipython()
ip.register_magics(CustomMagics)
Since notebook 5.1 you can use a new tag: raises-exception
This will indicate that exception in the specific cell is expected and jupyter will not stop the execution.
(In order to set a tag you have to choose from the main menu: View -> Cell Toolbar -> Tags)
I have a few long-running experiments in my Jupyter Notebooks. Because I don’t know when they will finish, I add an email function to the last cell of the notebook, so I automatically get an email, when the notebook is done.
But when there is a random exception in one of the cells, the whole notebook stops executing and I never get any email. So I’m wondering if there is some magic function that could execute a function in case of an exception / kernel stop.
Like
def handle_exception(stacktrace):
send_mail_to_myself(stacktrace)
%%in_case_of_notebook_exception handle_exception # <--- this is what I'm looking for
The other option would be to encapsulate every cell in try-catch, right? But that’s soooo tedious.
Thanks in advance for any suggestions.
I don’t think there is an out-of-the-box way to do that not using a try..except statement in your cells. AFAIK a 4 years old issue mentions this, but is still in open status.
However, the runtools extension may do the trick.
Such a magic command does not exist, but you can write it yourself.
from IPython.core.magic import register_cell_magic
@register_cell_magic('handle')
def handle(line, cell):
try:
exec(cell)
except Exception as e:
send_mail_to_myself(e)
raise # if you want the full trace-back in the notebook
It is not possible to load the magic command for the entire notebook automatically, you have to add it at each cell where you need this feature.
%%handle
some_code()
raise ValueError('this exception will be caught by the magic command')
@show0k gave the correct answer to my question (in regards to magic methods). Thanks a lot! 🙂
That answer inspired me to dig a little deeper and I came across an IPython method that lets you define a custom exception handler for the whole notebook.
I got it to work like this:
from IPython.core.ultratb import AutoFormattedTB
# initialize the formatter for making the tracebacks into strings
itb = AutoFormattedTB(mode = 'Plain', tb_offset = 1)
# this function will be called on exceptions in any cell
def custom_exc(shell, etype, evalue, tb, tb_offset=None):
# still show the error within the notebook, don't just swallow it
shell.showtraceback((etype, evalue, tb), tb_offset=tb_offset)
# grab the traceback and make it into a list of strings
stb = itb.structured_traceback(etype, evalue, tb)
sstb = itb.stb2text(stb)
print (sstb) # <--- this is the variable with the traceback string
print ("sending mail")
send_mail_to_myself(sstb)
# this registers a custom exception handler for the whole current notebook
get_ipython().set_custom_exc((Exception,), custom_exc)
So this can be put into a single cell at the top of any notebook and as a result it will do the mailing in case something goes wrong.
Note to self / TODO: make this snippet into a little python module that can be imported into a notebook and activated via line magic.
Be careful though. The documentation contains a warning for this set_custom_exc method: “WARNING: by putting in your own exception handler into IPython’s main execution loop, you run a very good chance of nasty crashes. This facility should only be used if you really know what you are doing.”
Why exec is not always the solution
It’s some years later and I had a similar issue trying to handle errors with Jupyter magics. However, I needed variables to persist in the actual Jupyter notebook.
%%try_except print
a = 12
raise ValueError('test')
In this example, I want the error to print (but could be anything such as e-mail as in the opening post), but also a == 12 to be true in the next cell. For that reason, the method exec suggested does not work when you define the magic in a different file. The solution I found is to use the IPython functionalities.
How you can solve it
from IPython.core.magic import line_magic, cell_magic, line_cell_magic, Magics, magics_class
@magics_class
class CustomMagics(Magics):
@cell_magic
def try_except(self, line, cell):
""" This magic wraps a cell in try_except functionality """
try:
self.shell.ex(cell) # This executes the cell in the current namespace
except Exception as e:
if ip.ev(f'callable({how})'): # check we have a callable handler
self.shell.user_ns['error'] = e # add error to namespace
ip.ev(f'{how}(error)') # call the handler with the error
else:
raise e
# Register
from IPython import get_ipython
ip = get_ipython()
ip.register_magics(CustomMagics)
Since notebook 5.1 you can use a new tag: raises-exception
This will indicate that exception in the specific cell is expected and jupyter will not stop the execution.
(In order to set a tag you have to choose from the main menu: View -> Cell Toolbar -> Tags)