If condition is True run all cells below in a Jupyter Notebook
Question:
I am creating a basic ETL script in Jupyter Notebook that refreshes data daily in a Google Sheet. However, sometimes the database is not updated on it’s scheduled time due to a failed load, etc.
I have created this line to verify that the date of the db refresh is today’s date
if pipeline_detail['data_refresh_date'][0] == datetime.date.today():
If this condition is True I want the code to continue running and all of the below cells to execute. Is there a way to do this in Jupyter Notebook?
Answers:
By default at this time, notebooks kicked off with ‘Run All’ or ‘Restart and Run All’ cease running if you hit an error, and so you can put an error to encounter in yourself. With a modified version of your condition test, i.e., testing for the inverse, the error will only be encountered if the condition you want is met.
Example
Imagine this in a cell:
if a != 3:
raise ValueError("a cannot be 3")
print("all is good, and so continuing to run")
That will mean that if the value of a isn’t currently three then that cell will throw an error and everything ceases running at that point. If a is three then then it will print all is good because of the code in the cell after the conditional test and then continue on to process the next cells in the notebook.
Adapting the example for your case
Adapting that general example to your case, you’d have something along the lines this pseudo-code in your cell:
...initial code in this cell...
if pipeline_detail['data_refresh_date'][0] != datetime.date.today():
raise ValueError("the date of the db refresh is not today's date")
...rest of code for this cell..
I am creating a basic ETL script in Jupyter Notebook that refreshes data daily in a Google Sheet. However, sometimes the database is not updated on it’s scheduled time due to a failed load, etc.
I have created this line to verify that the date of the db refresh is today’s date
if pipeline_detail['data_refresh_date'][0] == datetime.date.today():
If this condition is True I want the code to continue running and all of the below cells to execute. Is there a way to do this in Jupyter Notebook?
By default at this time, notebooks kicked off with ‘Run All’ or ‘Restart and Run All’ cease running if you hit an error, and so you can put an error to encounter in yourself. With a modified version of your condition test, i.e., testing for the inverse, the error will only be encountered if the condition you want is met.
Example
Imagine this in a cell:
if a != 3:
raise ValueError("a cannot be 3")
print("all is good, and so continuing to run")
That will mean that if the value of a isn’t currently three then that cell will throw an error and everything ceases running at that point. If a is three then then it will print all is good because of the code in the cell after the conditional test and then continue on to process the next cells in the notebook.
Adapting the example for your case
Adapting that general example to your case, you’d have something along the lines this pseudo-code in your cell:
...initial code in this cell...
if pipeline_detail['data_refresh_date'][0] != datetime.date.today():
raise ValueError("the date of the db refresh is not today's date")
...rest of code for this cell..