How to run python code in the currently activated virtualenv with subprocess?
Question:
I have a virtualenv named 'venv' and it is activate:
(venv)>
and I wrote codes that I’ll run it in the virtualenv (main.py):
import subprocess
result = subprocess.run('python other.py', stdout=subprocess.PIPE)
but when I run main.py file:
(venv)> python main.py
subprocess does not execute the command (python other.py) in the virtualenv i.e venv
How to run subprocess command in the current virtualenv session?
Answers:
A child process can’t run commands in its parent process without that process’s involvement.
This is why ssh-agent requires usage as eval "$(ssh-agent -s)" to invoke the shell commands it emits on output, for example. Thus, the literal thing you’re asking for here is impossible.
Fortunately, it’s also unnecessary.
virtualenvs use environment variables inherited by child processes.
This means that you don’t actually need to use the same shell that has a virtualenv activated to start a new Python interpreter intended to use the interpreter/libraries/etc. from that virtualenv.
subprocess.run must be passed a list, or shell=True must be used.
Either do this (which is better!)
import subprocess
result = subprocess.run(['python', 'other.py'], stdout=subprocess.PIPE)
Or this (which is worse!)
import subprocess
result = subprocess.run('python other.py', stdout=subprocess.PIPE, shell=True)
If you want to run a script with the same Python executable being used to run the current script, don’t use python and rely on the path being set up properly, just use sys.executable:
A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense.
This works if you executed the script with python myscript.py relying on the active virtualenv’s PATH. It also works if you executed the script with /usr/local/bin/python3.6 to ignore the PATH and test your script with a specific interpreter. Or if you executed the script with myscript.py, relying on a shbang line created at installation time by setuptools. Or if the script was run as a CGI depending on your Apache configuration. Or if you sudod the executable, or did something else that scraped down your environment. Or almost anything else imaginable.1
As explained in Charles Duffy’s answer, you still need to use a list of arguments instead of a string (or use shell=True, but you rarely want to do that). So:
result = subprocess.run([sys.executable, 'other.py'], stdout=subprocess.PIPE)
1. Well, not quite… Examples of where it doesn’t work include custom C programs that embed a CPython interpreter, some smartphone mini-Python environments, old-school Amiga Python, … The one most likely to affect you—and it’s a pretty big stretch—is that on some *nix platforms, if you write a program that execs Python by passing incompatible names for the process and arg0, sys.executable can end up wrong.
I have a virtualenv named 'venv' and it is activate:
(venv)>
and I wrote codes that I’ll run it in the virtualenv (main.py):
import subprocess
result = subprocess.run('python other.py', stdout=subprocess.PIPE)
but when I run main.py file:
(venv)> python main.py
subprocess does not execute the command (python other.py) in the virtualenv i.e venv
How to run subprocess command in the current virtualenv session?
A child process can’t run commands in its parent process without that process’s involvement.
This is why ssh-agent requires usage as eval "$(ssh-agent -s)" to invoke the shell commands it emits on output, for example. Thus, the literal thing you’re asking for here is impossible.
Fortunately, it’s also unnecessary.
virtualenvs use environment variables inherited by child processes.
This means that you don’t actually need to use the same shell that has a virtualenv activated to start a new Python interpreter intended to use the interpreter/libraries/etc. from that virtualenv.
subprocess.run must be passed a list, or shell=True must be used.
Either do this (which is better!)
import subprocess
result = subprocess.run(['python', 'other.py'], stdout=subprocess.PIPE)
Or this (which is worse!)
import subprocess
result = subprocess.run('python other.py', stdout=subprocess.PIPE, shell=True)
If you want to run a script with the same Python executable being used to run the current script, don’t use python and rely on the path being set up properly, just use sys.executable:
A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense.
This works if you executed the script with python myscript.py relying on the active virtualenv’s PATH. It also works if you executed the script with /usr/local/bin/python3.6 to ignore the PATH and test your script with a specific interpreter. Or if you executed the script with myscript.py, relying on a shbang line created at installation time by setuptools. Or if the script was run as a CGI depending on your Apache configuration. Or if you sudod the executable, or did something else that scraped down your environment. Or almost anything else imaginable.1
As explained in Charles Duffy’s answer, you still need to use a list of arguments instead of a string (or use shell=True, but you rarely want to do that). So:
result = subprocess.run([sys.executable, 'other.py'], stdout=subprocess.PIPE)
1. Well, not quite… Examples of where it doesn’t work include custom C programs that embed a CPython interpreter, some smartphone mini-Python environments, old-school Amiga Python, … The one most likely to affect you—and it’s a pretty big stretch—is that on some *nix platforms, if you write a program that execs Python by passing incompatible names for the process and arg0, sys.executable can end up wrong.