PySpark: How can I suppress %run output in PySpark cell when importing variables from another Notebook?
Question:
I am using multiple notebooks in PySpark and import variables across these notebooks using %run path. Every time I run the command, all variables that I displayed in the original notebook are being displayed again in the current notebook (the notebook in which I %run). But I do not want them to be displayed in the current notebook. I only want to be able to work with the imported variables. How do I suppress the output being display every time? Note, I am not sure if it matters, but I am working in DataBricks. Thank you!
Command example:
%run /Users/myemail/Nodebook
Answers:
This is expected behaviour, when you use %run command allows you to include another notebook within a notebook. This command lets you concatenate various notebooks that represent key ETL steps, Spark analysis steps, or ad-hoc exploration. However, it lacks the ability to build more complex data pipelines.
Notebook workflows are a complement to %run because they let you return values from a notebook. This allows you to easily build complex workflows and pipelines with dependencies. You can properly parameterize runs (for example, get a list of files in a directory and pass the names to another notebook—something that’s not possible with %run) and also create if/then/else workflows based on return values. Notebook workflows allow you to call other notebooks via relative paths.
You implement notebook workflows with dbutils.notebook methods. These methods, like all of the dbutils APIs, are available only in Scala and Python. However, you can use dbutils.notebook.run to invoke an R notebook.
For more details, refer “Databricks – Notebook workflows“.
You can use the "Hide Result" option in the upper right toggle of the cell:
The correct answer is stated in the comments of one of the answers ("you can’t"). I ran into the problem that markdown titles (which I use to divide a notebook into sections) were shown when the notebook was %run in another notebook.
I found a workaround where you can keep headings in your notebook to be loaded. Instead of using %md markdown you can give cells a title by using the dropdown menu on the top right of the cell and choose "show title", this will give your cell a title like this:
And this text doesn’t show up when you import the notebook using %run in another notebook.
I am using multiple notebooks in PySpark and import variables across these notebooks using %run path. Every time I run the command, all variables that I displayed in the original notebook are being displayed again in the current notebook (the notebook in which I %run). But I do not want them to be displayed in the current notebook. I only want to be able to work with the imported variables. How do I suppress the output being display every time? Note, I am not sure if it matters, but I am working in DataBricks. Thank you!
Command example:
%run /Users/myemail/Nodebook
This is expected behaviour, when you use %run command allows you to include another notebook within a notebook. This command lets you concatenate various notebooks that represent key ETL steps, Spark analysis steps, or ad-hoc exploration. However, it lacks the ability to build more complex data pipelines.
Notebook workflows are a complement to %run because they let you return values from a notebook. This allows you to easily build complex workflows and pipelines with dependencies. You can properly parameterize runs (for example, get a list of files in a directory and pass the names to another notebook—something that’s not possible with %run) and also create if/then/else workflows based on return values. Notebook workflows allow you to call other notebooks via relative paths.
You implement notebook workflows with dbutils.notebook methods. These methods, like all of the dbutils APIs, are available only in Scala and Python. However, you can use dbutils.notebook.run to invoke an R notebook.
For more details, refer “Databricks – Notebook workflows“.
You can use the "Hide Result" option in the upper right toggle of the cell:
The correct answer is stated in the comments of one of the answers ("you can’t"). I ran into the problem that markdown titles (which I use to divide a notebook into sections) were shown when the notebook was %run in another notebook.
I found a workaround where you can keep headings in your notebook to be loaded. Instead of using %md markdown you can give cells a title by using the dropdown menu on the top right of the cell and choose "show title", this will give your cell a title like this:
And this text doesn’t show up when you import the notebook using %run in another notebook.