Running subprocess within different virtualenv with python
Question:
Let’s say I have 2 different versions of my app installed in 2 different virtualenvironments. myapp v1.0 and myapp v2.0.
Now I would like to compare those. The comparison is written in python itself. What would be the best way to do that? Let’s assume I can run them separately and both write an output file, which I can compare later.
One way to do that would be to write a bash script (that’s what I have currently). I activate one virtualenv, run myapp v1.0, activate another virtualenv, run myapp v2.0. Later run a comparison module on those files. But I would like to add some more dynamics there (take some optional arguments etc) which would be easier with python.
Edit:
Currently I have something like that (a bash script):
source virtualenv1/bin/activate
python my_script.py
deactivate
source virtualenv2/bin/activate
python my_other_script.py
deactivate
python my_comparison_script.py
instead, I’d like to do only:
python my_comparison_script.py
and my scripts would be run inside this.
Answers:
what exactly is the question? how to use subprocess to execute shell commands? if that’s the case, some simplistic pseudo code might look like:
import subprocess
myProcess = subprocess.Popen( ['these', 'are', 'for', 'the', 'shell'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
[outStream, errStream] = myProcess.communicate()
then you can do whatever you’d like with standard out (outStream) and do different things if errStream exists (standard error). this includes writing the standard out or standard error to a file. then i presume you would diff those files?
an actual code example (assuming you have python 2.6+ on a linux system) might look like:
import subprocess
with open('dateHelp.log', 'w') as dateLog:
with open('dateHelp.err', 'w') as errLog:
dateHelp = subprocess.Popen([ 'date', '-h'], stdout=dateLog,
stderr=errLog)
dateHelp.communicate()
The accepted answer does not address the problem of ‘activating’ a virtualenv in a subprocess.
If you start your application with a call to the python executable, like in your example it is actually very simple: you only have to explicitly point to the executable in the virtualenv.
import subprocess
subprocess.Popen(["virtualenv1/bin/python", "my_script.py"])
subprocess.Popen(["virtualenv2/bin/python", "my_other_script.py"])
will start the processes in the respective virtualenvs.
Important
To address the concerns voiced in the comments:
If you want to run a subprocess and be sure to use the same interpreter that the current process is running in you have to use sys.executable. Also available: sys.exec_prefix to access the site-specific directory prefix where the platform-dependent Python files are installed.
If you want a much more in depth discussion of this topic, have a look at this pull request.
A simple option would be to run a series of commands with subprocess as follows (note that shell=True is risky and should only be used if you can control the input).
import subprocess
cmd = 'source activate my_virtualenv; python my_script.py'
subprocess.call(cmd, shell=True, executable='/bin/bash')
And repeat as needed.
I think virtualenv documentation explains it nicely.
TL;DR
Runnig python venv binary directly is NOT the same as activating venv.
You also have to change PATH and VIRTUAL_ENV variables accordingly (look at os.environ)
$ source /path/to/ENV/bin/activate
This will change your $PATH so its first entry is the virtualenv’s
bin/ directory. (You have to use source because it changes your shell
environment in-place.) This is all it does; it’s purely a convenience.
If you directly run a script or the python interpreter from the
virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or
/path/to/ENV/bin/python-script.py) then sys.path will automatically be
set to use the Python libraries associated with the virtualenv. But,
unlike the activation scripts, the environment variables PATH and
VIRTUAL_ENV will not be modified. This means that if your Python
script uses e.g. subprocess to run another Python script (e.g. via a
!/usr/bin/env python shebang line) the second script may not be executed with the same Python binary as the first nor have the same
libraries available to it. To avoid this happening your first script
will need to modify the environment variables in the same manner as
the activation scripts, before the second script is executed.
Let’s say I have 2 different versions of my app installed in 2 different virtualenvironments. myapp v1.0 and myapp v2.0.
Now I would like to compare those. The comparison is written in python itself. What would be the best way to do that? Let’s assume I can run them separately and both write an output file, which I can compare later.
One way to do that would be to write a bash script (that’s what I have currently). I activate one virtualenv, run myapp v1.0, activate another virtualenv, run myapp v2.0. Later run a comparison module on those files. But I would like to add some more dynamics there (take some optional arguments etc) which would be easier with python.
Edit:
Currently I have something like that (a bash script):
source virtualenv1/bin/activate
python my_script.py
deactivate
source virtualenv2/bin/activate
python my_other_script.py
deactivate
python my_comparison_script.py
instead, I’d like to do only:
python my_comparison_script.py
and my scripts would be run inside this.
what exactly is the question? how to use subprocess to execute shell commands? if that’s the case, some simplistic pseudo code might look like:
import subprocess
myProcess = subprocess.Popen( ['these', 'are', 'for', 'the', 'shell'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
[outStream, errStream] = myProcess.communicate()
then you can do whatever you’d like with standard out (outStream) and do different things if errStream exists (standard error). this includes writing the standard out or standard error to a file. then i presume you would diff those files?
an actual code example (assuming you have python 2.6+ on a linux system) might look like:
import subprocess
with open('dateHelp.log', 'w') as dateLog:
with open('dateHelp.err', 'w') as errLog:
dateHelp = subprocess.Popen([ 'date', '-h'], stdout=dateLog,
stderr=errLog)
dateHelp.communicate()
The accepted answer does not address the problem of ‘activating’ a virtualenv in a subprocess.
If you start your application with a call to the python executable, like in your example it is actually very simple: you only have to explicitly point to the executable in the virtualenv.
import subprocess
subprocess.Popen(["virtualenv1/bin/python", "my_script.py"])
subprocess.Popen(["virtualenv2/bin/python", "my_other_script.py"])
will start the processes in the respective virtualenvs.
Important
To address the concerns voiced in the comments:
If you want to run a subprocess and be sure to use the same interpreter that the current process is running in you have to use sys.executable. Also available: sys.exec_prefix to access the site-specific directory prefix where the platform-dependent Python files are installed.
If you want a much more in depth discussion of this topic, have a look at this pull request.
A simple option would be to run a series of commands with subprocess as follows (note that shell=True is risky and should only be used if you can control the input).
import subprocess
cmd = 'source activate my_virtualenv; python my_script.py'
subprocess.call(cmd, shell=True, executable='/bin/bash')
And repeat as needed.
I think virtualenv documentation explains it nicely.
TL;DR
Runnig python venv binary directly is NOT the same as activating venv.
You also have to change PATH and VIRTUAL_ENV variables accordingly (look at os.environ)
$ source /path/to/ENV/bin/activate
This will change your $PATH so its first entry is the virtualenv’s
bin/ directory. (You have to use source because it changes your shell
environment in-place.) This is all it does; it’s purely a convenience.If you directly run a script or the python interpreter from the
virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or
/path/to/ENV/bin/python-script.py) then sys.path will automatically be
set to use the Python libraries associated with the virtualenv. But,
unlike the activation scripts, the environment variables PATH and
VIRTUAL_ENV will not be modified. This means that if your Python
script uses e.g. subprocess to run another Python script (e.g. via a
!/usr/bin/env python shebang line) the second script may not be executed with the same Python binary as the first nor have the same
libraries available to it. To avoid this happening your first script
will need to modify the environment variables in the same manner as
the activation scripts, before the second script is executed.