Why are constants not accessible from a python file?
Question:
In my Databricks project, I have a very basic notebook which contains some constants as below:
RAW_FOLDER_PATH = 'dbfs:/mnt/formuleinsstorage/rawdata/unziped/'
PROCESSED_FOLDER_PATH = 'dbfs:/mnt/formuleinsstorage/processeddata'
MESSAGE_TO_WHEN_COMPLETING_NOTEBOOK_SUCCESSFULLY = 'Success'
dbutils.notebook.exit(MESSAGE_TO_WHEN_COMPLETING_NOTEBOOK_SUCCESSFULLY)
and then i need to run this notebook as part of another notebook using this code:
dbutils.notebook.run("./../helpers/configuration", 0)
dbutils.notebook.run("./../helpers/functions", 0)
Although, it looks like the dbutils.notebook.run() in the second notebook is successful, but when i try to get the value of RAW_FOLDER_PATH in the second notebook it gives me the below error:
NameError: name 'RAW_FOLDER_PATH' is not defined
I tried to run the same code using :
%run "./../helpers/configuration" and it was successful.
But, if i try to run 2 magic commands in one cell, such as below:
%run "./../helpers/configuration"
%run "./../helpers/configuration"
it fails with the below error:
Failed to parse %run command: string matching regex `$[w_]+' expected but `%' found. If notebook path contains spaces, wrap with double quotes.
Stacktrace:
/ingestions/ingest_circuite_file: python
So, how can i run 2 magic commands in 1 cell ?
Answers:
See the section "Comparison of %run and notebook workflows" here. In particular, dbutils.notebook.run() runs the other notebook in a new job, so it gets executed but not in the same kernel as the calling notebook. If you want the second notebook’s variables to be visible from the calling notebook, you should use %run instead, as in %run ./../helpers/configuration.
Edit in response to comment: You’ll need to put each %run command in a separate cell. Per the Databricks documentation: "%run must be in a cell by itself, because it runs the entire notebook inline."
In my Databricks project, I have a very basic notebook which contains some constants as below:
RAW_FOLDER_PATH = 'dbfs:/mnt/formuleinsstorage/rawdata/unziped/'
PROCESSED_FOLDER_PATH = 'dbfs:/mnt/formuleinsstorage/processeddata'
MESSAGE_TO_WHEN_COMPLETING_NOTEBOOK_SUCCESSFULLY = 'Success'
dbutils.notebook.exit(MESSAGE_TO_WHEN_COMPLETING_NOTEBOOK_SUCCESSFULLY)
and then i need to run this notebook as part of another notebook using this code:
dbutils.notebook.run("./../helpers/configuration", 0)
dbutils.notebook.run("./../helpers/functions", 0)
Although, it looks like the dbutils.notebook.run() in the second notebook is successful, but when i try to get the value of RAW_FOLDER_PATH in the second notebook it gives me the below error:
NameError: name 'RAW_FOLDER_PATH' is not defined
I tried to run the same code using :
%run "./../helpers/configuration" and it was successful.
But, if i try to run 2 magic commands in one cell, such as below:
%run "./../helpers/configuration"
%run "./../helpers/configuration"
it fails with the below error:
Failed to parse %run command: string matching regex `$[w_]+' expected but `%' found. If notebook path contains spaces, wrap with double quotes.
Stacktrace:
/ingestions/ingest_circuite_file: python
So, how can i run 2 magic commands in 1 cell ?
See the section "Comparison of %run and notebook workflows" here. In particular, dbutils.notebook.run() runs the other notebook in a new job, so it gets executed but not in the same kernel as the calling notebook. If you want the second notebook’s variables to be visible from the calling notebook, you should use %run instead, as in %run ./../helpers/configuration.
Edit in response to comment: You’ll need to put each %run command in a separate cell. Per the Databricks documentation: "%run must be in a cell by itself, because it runs the entire notebook inline."