How to set env variable in Jupyter notebook

You can setup environment variables in your code as follows:

import sys,os,os.path
sys.path.append(os.path.expanduser('~/code/eol_hsrl_python'))
os.environ['HSRL_INSTRUMENT']='gvhsrl'
os.environ['HSRL_CONFIG']=os.path.expanduser('~/hsrl_config')

This if of course a temporary fix, to get a permanent one, you probably need to export the variables into your ~.profile, more information can be found here

If you are using systemd I just found out that you seem to have to add them to the systemd unit file. This on Ubuntu 16. Putting them into the .profile and .bashrc (even the /etc/profile) resulted in the ENV Vars not being available in the juypter notebooks.

I had to edit:

/lib/systemd/system/jupyer-notebook.service

and put in the variable i wanted to read in the unit file like:

Environment=MYOWN_VAR=theVar

and only then could I read it from within juypter notebook.

If you need the variable set before you’re starting the notebook, the only solution which worked for me was env VARIABLE=$VARIABLE jupyter notebook with export VARIABLE=value in .bashrc.

In my case tensorflow needs the exported variable for successful importing it in a notebook.

A related (short-term) solution is to store your environment variables in a single file, with a predictable format, that can be sourced when starting a terminal and/or read into the notebook. For example, I have a file, .env, that has my environment variable definitions in the format VARIABLE_NAME=VARIABLE_VALUE (no blank lines or extra spaces). You can source this file in the .bashrc or .bash_profile files when beginning a new terminal session and you can read this into a notebook with something like,

import os
env_vars = !cat ../script/.env
for var in env_vars:
    key, value = var.split('=')
    os.environ[key] = value

I used a relative path to show that this .env file can live anywhere and be referenced relative to the directory containing the notebook file. This also has the advantage of not displaying the variable values within your code anywhere.

A gotcha I ran into: The following two commands are equivalent. Note the first cannot use quotes. Somewhat counterintuitively, quoting the string when using %env VAR ... will result in the quotes being included as part of the variable’s value, which is probably not what you want.

%env MYPATH=C:/Folder Name/file.txt

and

import os
os.environ['MYPATH'] = "C:/Folder Name/file.txt"

You can also set the variables in your kernel.json file:

My solution is useful if you need the same environment variables every time you start a jupyter kernel, especially if you have multiple sets of environment variables for different tasks.

To create a new ipython kernel with your environment variables, do the following:

• Read the documentation at https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs
• Run jupyter kernelspec list to see a list with installed kernels and where the files are stored.
• Copy the directory that contains the kernel.json (e.g. named python2) to a new directory (e.g. python2_myENV).
• Change the display_name in the new kernel.json file.
• Add a env dictionary defining the environment variables.

Your kernel json could look like this (I did not modify anything from the installed kernel.json except display_name and env):

{
 "display_name": "Python 2 with environment",
 "language": "python",
 "argv": [
  "/usr/bin/python2",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "env": {"LD_LIBRARY_PATH":""}
}

Use cases and advantages of this approach

• In my use-case, I wanted to set the variable LD_LIBRARY_PATH which effects how compiled modules (e.g. written in C) are loaded. Setting this variable using %set_env did not work.
• I can have multiple python kernels with different environments.
• To change the environment, I only have to switch/ restart the kernel, but I do not have to restart the jupyter instance (useful, if I do not want to loose the variables in another notebook). See -however – https://github.com/jupyter/notebook/issues/2647

If you’re using Python, you can define your environment variables in a .env file and load them from within a Jupyter notebook using python-dotenv.

Install python-dotenv:

pip install python-dotenv

Load the .env file in a Jupyter notebook:

%load_ext dotenv
%dotenv

If your notebook is being spawned by a Jupyter Hub, you might need to configure (in jupyterhub_config.py) the list of environment variables that are allowed to be carried over from the JupyterHub process environment to the Notebook environment by setting

c.Spawner.env_keep = [VAR1, VAR2, ...]

(https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.env_keep)

See also: Spawner.environment

you can run jupyter notebook with docker and don(t have to manage dependancy leaks.

docker run -p 8888:8888 -v /home/mee/myfolder:/home/jovyan --name notebook1 jupyter/notebook
docker exec -it notebook1 /bin/bash

then kindly ask jupyter about the opened notebooks,
jupyter notebook list
http:// 0.0.0.0:8888/?token=012456788997977a6eb11e45fffff
Url can be copypasted, verify port if you have changed it.

Create a notebook and paste the following,
into the notebook

!pip install python-dotenv
import dotenv
%load_ext dotenv
%dotenv

if you have vsCode just create the env like always in the terminal then go to the jupyter file then click select kernel in the top right side then the command palette will open with the env path just click on it and it will take the env.

I had to run !export CUDA_VISIBLE_DEVICES=0,1.

Strange that there is no answer on this since this was the only way to get some ParallelData code to work, and I am not the only one with this. It took me many hours of trial and error that the answers here did not show the clearly needed:

• !export ....
• You can also run it in a CLI execution like with CUDA_VISIBLE_DEVICES=4,5,6,7 python forward_test_ins.py.

The main thing is that you should not take os.environ[] or %set_env into the code, see for example:

• How to Solve "CUDA: Invalid Device Ordinal" Error in PyTorch Single-GPU Inference on a Model Trained with DataParallel? that proves it for both os.environ[] and %set_env with code,
• this CUDA_VISIBLE_DEVICES example for parallel computing with many GPUs at Transformers Trainer: "RuntimeError: module must have its parameters … on device cuda:6 (device_ids[0]) but found one of them on device: cuda:0",
• the entries at setting CUDA_VISIBLE_DEVICES just has no effect #9158. If I recall it right, the magic command of the accepted answer also was not enough to steer the parallel computing.