Using subprocess in anaconda environment

Question

I use Python 3.6.6 :: Anaconda, Inc.

I try to use subprocess to call other python script.

subprocess.run("python -V", shell=True)

I tried this code but outcome is

Python 2.7.12

My local python get caught

I also tried

subprocess.run("bash -c 'source /home/hclee/anaconda3/envs/calamari/lib/python3.6/venv/scripts/common/activate
/home/hclee/anaconda3/envs/calamari && python -V && source deactivate'", shell=True)

but I got the same result

Asked By: hochan

||

Source

Answer 1

Run source activate root in linux, or activate root in Windows to activate the environment before running your code.

If this does not help you, try for a quick and dirty fix e.g.:

subprocess.run('bash -c "source activate root; python -V"', shell=True)

The reason you need to call bash is that python’s run will not call the bash environment, but another which is a bit more constrained and does not contain source, so here we need to call bash… But as mentioned, if you need this command, either you are doing something special, or something is wrong with your environment…

deactivate is not needed, it does nothing cause the shell it was run on will be destroyed…

Note: For newest conda versions, the provided code will work, but there are also these options that work similarly:

conda deactivate:

subprocess.run('bash -c "conda deactivate; python -V"', shell=True)

conda activate root or base:

subprocess.run('bash -c "conda activate root; python -V"', shell=True)
subprocess.run('bash -c "conda activate base; python -V"', shell=True)

Answered By: ntg

Answer 2

I don’t think sourcing a conda env in every subprocess call of your code is a good idea.

Instead you can find the bin dir of your current sourced env and grab the full path to binaries from there. Then pass these to subprocess when you want to call them.

import os
import sys
import subprocess

# what conda env am I in (e.g., where is my Python process from)?
ENVBIN = sys.exec_prefix

# what binaries am I looking for that are installed in this env?
BIN1 = os.path.join(ENVBIN, "bin", "ipython")
BIN2 = os.path.join(ENVBIN, "bin", "python")
BIN3 = os.path.join(ENVBIN, "bin", "aws")

# let's make sure they exist, no typos.
for bin in (BIN1, BIN2, BIN3):
    assert os.path.exists(bin), "missing binary {} in env {}".format(bin, ENVBIN)

# then use their full paths when making subprocess calls
for bin in (BIN1, BIN2, BIN3):
    cmd = ["which", bin]
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    print(proc.communicate()[0].decode())

The printed results show that subprocess is using the binaries from the conda environment instead of from the base (default) env.

.../miniconda3/envs/aws/bin/ipython
.../miniconda3/envs/aws/bin/python
.../miniconda3/envs/aws/bin/aws

Answered By: Flower-doctor

Answer 3

If your console/notebook is already using the correct environment, you can call subprocess with sys.executable to use current environment:

import sys
import subprocess
subprocess.run(f"{sys.executable} -V", shell=True)

Answered By: ronkov

Answer 4

The above solutions didn’t work for me.
What did work is:

import subprocess
output = subprocess.run('''python -c "import sys; print(sys.executable)"
                  source <your-conda-base-dir>/etc/profile.d/conda.sh
                  conda activate <name-of-the-conda-environment-you-want-to-activate>
                  python -c "import sys; print(sys.executable)"
                  conda env export''', executable='/bin/bash', shell=True, capture_output=True)
print(bytes.decode(output.stdout))

Substitute <your-conda-base-dir> by the output of which conda executed in a the interactive terminal you normally use for executing conda commands, but remove /condabin/conda from the end of the output you get, such that the /etc/profile.d/conda.sh location will be found when the above subprocess.run() is executing.

Answered By: Sander Vanden Hautte

Using subprocess in anaconda environment

Question:

Answers: