jupyter notebook running kernel in different env

Question:

I’ve gotten myself into some kind of horrible virtualenv mess. Help?!

I manage environments with conda. Until recently, I only had a python2 jupyter notebook kernel, but I decided to drag myself kicking and screaming into the 21st century and installed a python3 kernel; I forget how I did it.

My main (anaconda) python defaults to 2.7.

So here I am, merrily trying to use beautiful soup from inside my shiny new python3 kernel, and I don’t seem to be able to do anything to get at whatever environment it’s finding packages in. Viz (all from notebook):

from bs4 import BeautifulSoup 

-> ImportError: No module named 'bs4'

Ok, fine, I’ll install it using shell magic. Right? Right?

! pip install bs4

--> Collecting bs4
  Downloading bs4-0.0.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in /Users/[MY-USER]/anaconda/lib/python2.7/site-packages (from bs4)
[...]
Successfully built bs4
Installing collected packages: bs4
Successfully installed bs4-0.0.1

from bs4 import BeautifulSoup 

-> ImportError: No module named 'bs4'

Oh no. Does it think I’m in a 2.7 env even though I’m running a python3 kernel? That won’t do.

! conda info --envs
--> # conda environments:
#
flaskenv                 /Users/[MY-USER]/anaconda/envs/flaskenv
mesa                     /Users/[MY-USER]/anaconda/envs/mesa
py35                     /Users/[MY-USER]/anaconda/envs/py35
root                  *  /Users/[MY-USER]/anaconda

Ok, I can fix that. One of those is a 3.5 env.

! source activate py35
--> prepending /Users/[MY-USER]/anaconda/envs/py35/bin to PATH

! conda install beautifulsoup4
--> Fetching package metadata .......
Solving package specifications: ..........

# All requested packages already installed.
# packages in environment at /Users/[MY-USER]/anaconda:
#
beautifulsoup4            4.4.1                    py27_0  

concerning…

! pip install bs4
--> Requirement already satisfied (use --upgrade to upgrade): bs4 in /Users/[MY-USER]/anaconda/lib/python2.7/site-packages

more concerning…

from bs4 import BeautifulSoup 

-> ImportError: No module named 'bs4'

ARRGH!!! headdesk Am I going to have to kill the kernel in order to fix this (and re-run a bit of work)? Is killing the kernel even going to work? How do I get my jupyter kernel to know what environment it’s supposed to be running under?

thanks!

Asked By: Paul Gowder

||

Answers:

This is a tricky part of ipython / Jupyter. The set of kernels available are independent of what your virtualenv is when you start jupyter Notebook. The trick is setting up the the ipykernel package in the environment you want to identify itself uniquely to jupyter. From docs on multiple ipykernels,

source activate ENVNAME
pip install ipykernel
python -m ipykernel install --user --name ENVNAME --display-name "Python (whatever you want to call it)"

If you only want to have a single Python 3 kernel, from the conda environment, just use python -m ipykernel install --user and it will reset the default python to the one in the virtualenv.

And yes, you will need to restart the kernel and re-run the prior steps.

See Also Using both Python 2.x and Python 3.x in IPython Notebook

Answered By: tschundler

@tschundler’s solution works perfectly if your environment has already been created.

If you want to change the default kernel at the creation of your virtual environment and avoid any manual configuration, you just need to add jupyter at the end of the conda command:

conda create --name ENVNAME python=PYTHONVERSION jupyter

The correct kernel will then be used when you use ipython or jupyter notebook.

Answered By: Antoine Dusséaux

In my case somehow jupyter wasn’t able to ‘pick’ the virtual environment’s python. So I had to edit ~/.local/share/jupyter/kernels/{my_env_name}/kernel.json
and add path to the interpreter

in the argv key.

Answered By: markroxor

There is also an easy way here

workon my-virtualenv-name  # activate your virtualenv, if you haven't already
pip install tornado==4.5.3
pip install ipykernel==4.8.2

You should now be able to see your kernel in the IPython notebook menu: Kernel -> Change kernel and be able to switch to it (you may need to refresh the page before it appears in the list). IPython will remember which kernel to use for that notebook from then on.

This worked for me. source

Answered By: Rakend Dubba
pip install --user ipykernel
python -m ipykernel install --user --name=myenv

Output
Installed kernelspec myenv in /home/user/.local/share/jupyter/kernels/myenv

and go to above directory
open kernel.json

{
 "argv": [
  "/home/user/anaconda3/envs/myenv/bin/python", # path to your virtual environment python
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "myenv",
 "language": "python"
}
Answered By: vishal

I know this is an old question, but nobody has mentioned nb_conda_kernels, which was made to solve this problem. You must have it installed in the environment from which you launched the running notebook, and any other environment which has kernel/s installed (ipykernel, r-irkernel, etc ) will be available as an option (Start Preferred Kernel):

enter image description here

Answered By: matrs

Sharing my experience here, in case someone made the same mistake:

TL;DR

In VSCode I created the virtual environment from an Ubuntu terminal but forgot to activate the WSL extension too. Once I activated the WSL extension, I was able to change the Jupyterkernel to my virtual environment.

Full explanation

Using VSCode I work with the WSL extension (see here). I created my virtual environment from an Ubuntu terminal [Created a new terminal and selected ‘Ubuntu (WSL)’] and then went to my Jupyter notebook to change my kernel, but it did not appear in the list.

I tried the solution proposed above of adding the path to kernel.json in my virtual environment, adapting to my case (I don’t use conda) but I was surprised to see that the folder bin in my environment did not have a file named python.

The solution was to activate the WSL extension: lower right corner > click on the arrows (Open a remote connection) > reopen folder in WSL.

After this, I could see in the explorer two new entries in the bin folder for Python and Python3 (the arrow on the right side seem to indicate it’s a symbolic link).
Virtual environment now shows the python symbolic links

Then I went to my Jupyter notebook and changed the kernel to my virtual environment.

In the end, I did not have to change the path in kernel.json.

P.S: The above is my understanding of my problem and the solution given my non-existing knowledge about Linux, so any comments to complement/improve this answer are very welcome!

Answered By: Jadelcar