Best way to execute a python script in a given conda environment

Question:

I want to execute a python script executed.py from another python script trigger.py using the subprocess package. The executed.py must be ran in a different conda environment than trigger.py (let say executed_env and trigger_env). What is the best way to do that ? My current code of trigger.py is:

command = "python executed.py --option1 -dir /path/to/dir"
args = shlex.split(command)
my_subprocess = subprocess.Popen(args)

It returns an error since executed.py is ran in the trigger_env environment.

Asked By: Robin

||
Source

Answers:

If you just need to use the other python then I believe you simply need to use the full path to the other python in your command.

Try going into your executed_env (i.e. source activate executed_env if Linux) and do which python. Let’s assume that returns HOME/.conda/envs/executed_env/bin/python now that becomes the python that you use in your command — i.e. command = "HOME/.conda/envs/executed_env/bin/python executed.py --option1 -dir /path/to/dir". This

For example, let’s run executed.py in a different python version.

  • Create your py27 environment conda create -n py27 python=2.7 this is the trigger_env.
  • Create your py35 environment conda create -n py35 python=3.5 this is the executed_env.
  • Get the full python path to py35 by running source activate py35 and then which python (let’s call that EXECUTED_PYTHON for this description). Deactivate via source deactivate.

  • Then we create executed.py

    import sys
print(sys.version)

  • Then the trigger.py (included the arguments but they are doing nothing)

    command = "EXECUTED_PYTHON executed.py --option1 -dir /path/to/dir"
args = shlex.split(command)
my_subprocess = subprocess.Popen(args)

  • Now let’s run it in the trigger_env — source activate py27 and python trigger.py.

  • This prints 3.5.2 |Continuum Analytics, Inc. [...] (3.5 being the important part). You can see it is using the other python in the executed_env.
Answered By: Paul

I did some research on any conda built-in methods and found the following. The first is a workaround, and the second the final solution built-in solution (not available at time of writing this post).

  1. conda-wrappers.
    Guilherme Melo created a wrappers for the python executables within a conda environment. If you set it as the python interpreter in your IDE, e.g. PyCharm, it will activate the conda environment from which it is called, and then call the python interpreter.
    Look here under section “Creating conda wrappers”:
    https://pypi.org/project/exec-wrappers/

  2. conda run.
    A long discussion on the conda github page on a standard and fast way to execute a command inside an environment led to the implementation of a new command (actually a re-invocation as it was available before):
    conda run

It is described in issue #7320 and will be released in conda-4.6 hopefully in October 2018!

Answered By: Agile Bean

Using conda on windows environment, I just copied the way conda does for starting different jupyter notebooks each from a different environment. All shortcuts call to "C:ProgramDataAnaconda3cwp.py" script using "C:ProgramDataAnaconda3python.exe", then with additional parameters you can choose the environment and the file to be executed.

In my case, environment name = AutomateXXReporting, file to be executed = C:Usersismael.serranoDocumentsGITxxxnotebooksexportsget_JIRA_user_streams.py

C:ProgramDataAnaconda3python.exe C:ProgramDataAnaconda3cwp.py C:Usersismael.serrano.condaenvsAutomateXXReporting C:Usersismael.serrano.condaenvsAutomateXXReportingpython.exe C:Usersismael.serranoDocumentsGITxxxnotebooksexportsget_JIRA_user_streams.py

Previously I had proceed successfully as Paul indicated, but reached a case where it was not sufficient. The python from the full path was not able to find some libraries, using cwp.py it worked.

Answered By: Ismael Serrano