How to test if Python is running from a virtualenv during package installation
Question:
I have a Python software which includes a configuration file and a manpage. To install these, I have the following line in my setup.py (as described on http://docs.python.org/2/distutils/setupscript.html#installing-additional-files):
data_files = [('/etc/foo', ['foo.conf']), ('/usr/share/man/man1', ['foo.1'])]
This works just fine when I want to install the software as root with python setup.py install, but of course fails in a virtualenv, as the user is not permitted to write to /etc and /usr/share/man.
What is the best practice to fix that? Check for VIRTUAL_ENV in the current environment and just not install those files at all? The software will look for foo.conf in the local directory, so that should be no problem. The user would miss the manpage, but there is no sane way to install it anyways, as man won’t look for it anywhere near the virtualenv.
Answers:
Ultimately it seems that your question is really about how to detect whether the running Python is in a virtualenv. To get to the bottom of this we must understand how virtualenv actually works.
When you run the activate script inside a virtualenv, it does two things:
- it updates the
PATH environment variable to include the bin directory from the virtualenv, so that when you run python the binary from the virtualenv will run.
- it sets a variable
VIRTUAL_ENV so that the activate script itself can keep track of the activation.
It’s perfectly acceptable to directly run the python from the virtualenv, and at runtime python does not use the VIRTUAL_ENV variable at all. Instead, it determines the directory containing the running pythonbinary and uses the parent directory as its “prefix”.
You can determine the system’s prefix by importing the sys module and consulting sys.prefix. However, it would be a bad idea to depend on the value of this when a virtualenv is not activated because this is a build-time setting for Python that can easily be customized, and it will definitely vary between platforms.
However, Python does have one slight runtime difference when it runs from a virtualenv prefix vs. its compiled-in prefix: the sys package has an additional variable real_prefix that returns the prefix that is compiled into the Python binary. Therefore one could use this to recognize that Python is running in a non-default location, which is reasonably likely to mean it’s running from a virtualenv:
import sys
if getattr(sys, "real_prefix", None) is not None:
print "Maybe in a virtualenv"
else:
print "Probably not in a virtualenv"
However, even this is not an exact science. All this really tells you is that the python binary is not at the location specified at compile time. It does not tell you whether the current user has access to write to /usr/share/man — there are some (possibly edge) cases where this won’t give you the right answer:
-
If the user has compiled his own Python from source in his home directory and its compiled prefix is /home/johnd/local-python then real_prefix won’t be set but the user still has write access to his Python lib directory, and probably not write access to /etc or /usr/share/man
-
Likewise, on some systems the administrator may have granted group-write privileges on /usr/lib/python2.7 to a certain group of app developers so that they can install Python modules, but not have granted them write access to other system files.
So I think in the end the best you can do is a heuristic and it may be better to instead just avoid using absolute paths in data_files for any module you expect to be used inside a virtualenv. A compromise might be to simply split your module into two distributions, one representing the localizable source files and another representing the system-wide configuration to make it run. The latter can depend on the former so that users can still easily install it, but those using virtualenv have the option of using the other former directly.
Narrowing the question to its title only:
"How to test if Python is running from a virtualenv during package installation"
Being more than 7 years old, the accepted answer did not work for me (python 3.6.9 on Ubuntu 18.04).
It failed to detect I’m in a virtualenv (activated and being used) when I was. More specifically, getattr(sys, "real_prefix", None) returned "/usr" and (wrongly) concluded I’m not running from a virtual-env.
The alternative solution as suggested in the question itself worked fine (1).
It is simple & straightforward, despite objections above that it might be unreliable under non-common situations.
Save to a file is-in-venv and make it executable, and it can be called from a shell, if is-in-venv; then ...; else ...; fi
#!/usr/bin/env python3
import os
import sys
venvdir = os.getenv('VIRTUAL_ENV')
if venvdir and os.path.isdir(venvdir):
print("python3 virtual-env detected: %s" % venvdir)
sys.exit(0)
else:
print("python3 not in a virtual env")
sys.exit(1)
(1) For the benefit of potential future users looking for a working solution to the title as written, for python3, circa 2020.
Many Python users erroneously believe that it’s the modifications to $PATH performed by the bin/activate script, that define a virtual env. But that’s not true — the activate script is a convenience that has no effect on Python at all, it merely configures your environment to access the venv more easily.
A Python virtual environment is defined by the changes that occur to sys.path when running inside it, and by Python’s automatic discovery of the library dir relative to the filesystem location of the python[3[.n]] interpreter executable. All of the process environment variables, including $PATH and $VIRTUAL_ENV, are irrelevant to the virtual environment within Python. Any detection that uses environment variables is really just a check for whether the activate script has been run. And as Martin has tried to point out, the activate script is not a prerequisite for running within a venv.
If you have a virtual environment at /home/me/my_venv/, sourcing /home/me/my_venv/bin/activate and then running python3 will get you a REPL that’s running inside that virtual environment. But so will running /home/me/my_venv/bin/python3 WITHOUT sourcing activate. The $PATH will be different and $VIRTUAL_ENV won’t be set, but sys.path will be identical using either method. (Try it and see!)
So, the key to detecting a venv lies with how sys.path changes when the interpreter is run from a location other than its installed path, and that starts with the modified sys.prefix.
sys.real_prefix is something the virtualenv package used to set. It caused problems, ironically because venv failed to detect that it was running inside a virtualenv, so virtualenv dropped sys.real_path with its v20 rewrite. In fact, as of v20 virtualenv creates its virtual environment using the standard library venv module if it’s available. (Which it is, in all currently supported Python versions.)
So, for all currently-supported Python releases (3.7–3.11), the following will effectively detect a virtual environment without relying on $PATH or $VIRTUAL_ENV. (I kept the sys.real_path check, even though it should rarely succeed and would indicate a very out-of-date virtualenv package.)
def in_venv() -> bool:
"""Determine whether Python is running from a venv."""
import sys
if hasattr(sys, 'real_prefix'):
return True
pfx = getattr(sys, 'base_prefix', sys.prefix)
return pfx != sys.prefix
That still won’t detect a very specific type of "virtual" environment, though: A vendored Python interpreter distribution that’s been installed completely separate from the system installation(s). Those won’t register as virtual environments, because the sys.base_path will be the same as sys.prefix —they’ll both point to the runtime’s install directory.
For example, ActiveState Python on Windows will install a vendored interpreter binary into each project instance that’s created on the system. That runtime may have a sys.prefix like c:users$nameappdatalocalactivestatecache$random, and sys.base_prefix will be the same. Some Node.js packages do similar things with vendored Python installations.
In these situations, sys.path will also be entirely located under the sys.prefix path, making the installation not technically a virtual environment at all. It’s merely a non-system, vendored Python runtime environment with no connection to or dependencies on any system runtime. Those environments are much trickier to reliably detect, and coding assumptions about what to do in those conditions is a much dicier prospect.
A very simple answer I’ve found for the question as it is, is to run:
pip -V # (Uppercase V)
I have a Python software which includes a configuration file and a manpage. To install these, I have the following line in my setup.py (as described on http://docs.python.org/2/distutils/setupscript.html#installing-additional-files):
data_files = [('/etc/foo', ['foo.conf']), ('/usr/share/man/man1', ['foo.1'])]
This works just fine when I want to install the software as root with python setup.py install, but of course fails in a virtualenv, as the user is not permitted to write to /etc and /usr/share/man.
What is the best practice to fix that? Check for VIRTUAL_ENV in the current environment and just not install those files at all? The software will look for foo.conf in the local directory, so that should be no problem. The user would miss the manpage, but there is no sane way to install it anyways, as man won’t look for it anywhere near the virtualenv.
Ultimately it seems that your question is really about how to detect whether the running Python is in a virtualenv. To get to the bottom of this we must understand how virtualenv actually works.
When you run the activate script inside a virtualenv, it does two things:
- it updates the
PATHenvironment variable to include thebindirectory from the virtualenv, so that when you runpythonthe binary from the virtualenv will run. - it sets a variable
VIRTUAL_ENVso that the activate script itself can keep track of the activation.
It’s perfectly acceptable to directly run the python from the virtualenv, and at runtime python does not use the VIRTUAL_ENV variable at all. Instead, it determines the directory containing the running pythonbinary and uses the parent directory as its “prefix”.
You can determine the system’s prefix by importing the sys module and consulting sys.prefix. However, it would be a bad idea to depend on the value of this when a virtualenv is not activated because this is a build-time setting for Python that can easily be customized, and it will definitely vary between platforms.
However, Python does have one slight runtime difference when it runs from a virtualenv prefix vs. its compiled-in prefix: the sys package has an additional variable real_prefix that returns the prefix that is compiled into the Python binary. Therefore one could use this to recognize that Python is running in a non-default location, which is reasonably likely to mean it’s running from a virtualenv:
import sys
if getattr(sys, "real_prefix", None) is not None:
print "Maybe in a virtualenv"
else:
print "Probably not in a virtualenv"
However, even this is not an exact science. All this really tells you is that the python binary is not at the location specified at compile time. It does not tell you whether the current user has access to write to /usr/share/man — there are some (possibly edge) cases where this won’t give you the right answer:
-
If the user has compiled his own Python from source in his home directory and its compiled prefix is
/home/johnd/local-pythonthenreal_prefixwon’t be set but the user still has write access to his Pythonlibdirectory, and probably not write access to/etcor/usr/share/man -
Likewise, on some systems the administrator may have granted group-write privileges on
/usr/lib/python2.7to a certain group of app developers so that they can install Python modules, but not have granted them write access to other system files.
So I think in the end the best you can do is a heuristic and it may be better to instead just avoid using absolute paths in data_files for any module you expect to be used inside a virtualenv. A compromise might be to simply split your module into two distributions, one representing the localizable source files and another representing the system-wide configuration to make it run. The latter can depend on the former so that users can still easily install it, but those using virtualenv have the option of using the other former directly.
Narrowing the question to its title only:
"How to test if Python is running from a virtualenv during package installation"
Being more than 7 years old, the accepted answer did not work for me (python 3.6.9 on Ubuntu 18.04).
It failed to detect I’m in a virtualenv (activated and being used) when I was. More specifically, getattr(sys, "real_prefix", None) returned "/usr" and (wrongly) concluded I’m not running from a virtual-env.
The alternative solution as suggested in the question itself worked fine (1).
It is simple & straightforward, despite objections above that it might be unreliable under non-common situations.
Save to a file is-in-venv and make it executable, and it can be called from a shell, if is-in-venv; then ...; else ...; fi
#!/usr/bin/env python3
import os
import sys
venvdir = os.getenv('VIRTUAL_ENV')
if venvdir and os.path.isdir(venvdir):
print("python3 virtual-env detected: %s" % venvdir)
sys.exit(0)
else:
print("python3 not in a virtual env")
sys.exit(1)
(1) For the benefit of potential future users looking for a working solution to the title as written, for python3, circa 2020.
Many Python users erroneously believe that it’s the modifications to $PATH performed by the bin/activate script, that define a virtual env. But that’s not true — the activate script is a convenience that has no effect on Python at all, it merely configures your environment to access the venv more easily.
A Python virtual environment is defined by the changes that occur to sys.path when running inside it, and by Python’s automatic discovery of the library dir relative to the filesystem location of the python[3[.n]] interpreter executable. All of the process environment variables, including $PATH and $VIRTUAL_ENV, are irrelevant to the virtual environment within Python. Any detection that uses environment variables is really just a check for whether the activate script has been run. And as Martin has tried to point out, the activate script is not a prerequisite for running within a venv.
If you have a virtual environment at /home/me/my_venv/, sourcing /home/me/my_venv/bin/activate and then running python3 will get you a REPL that’s running inside that virtual environment. But so will running /home/me/my_venv/bin/python3 WITHOUT sourcing activate. The $PATH will be different and $VIRTUAL_ENV won’t be set, but sys.path will be identical using either method. (Try it and see!)
So, the key to detecting a venv lies with how sys.path changes when the interpreter is run from a location other than its installed path, and that starts with the modified sys.prefix.
sys.real_prefix is something the virtualenv package used to set. It caused problems, ironically because venv failed to detect that it was running inside a virtualenv, so virtualenv dropped sys.real_path with its v20 rewrite. In fact, as of v20 virtualenv creates its virtual environment using the standard library venv module if it’s available. (Which it is, in all currently supported Python versions.)
So, for all currently-supported Python releases (3.7–3.11), the following will effectively detect a virtual environment without relying on $PATH or $VIRTUAL_ENV. (I kept the sys.real_path check, even though it should rarely succeed and would indicate a very out-of-date virtualenv package.)
def in_venv() -> bool:
"""Determine whether Python is running from a venv."""
import sys
if hasattr(sys, 'real_prefix'):
return True
pfx = getattr(sys, 'base_prefix', sys.prefix)
return pfx != sys.prefix
That still won’t detect a very specific type of "virtual" environment, though: A vendored Python interpreter distribution that’s been installed completely separate from the system installation(s). Those won’t register as virtual environments, because the sys.base_path will be the same as sys.prefix —they’ll both point to the runtime’s install directory.
For example, ActiveState Python on Windows will install a vendored interpreter binary into each project instance that’s created on the system. That runtime may have a sys.prefix like c:users$nameappdatalocalactivestatecache$random, and sys.base_prefix will be the same. Some Node.js packages do similar things with vendored Python installations.
In these situations, sys.path will also be entirely located under the sys.prefix path, making the installation not technically a virtual environment at all. It’s merely a non-system, vendored Python runtime environment with no connection to or dependencies on any system runtime. Those environments are much trickier to reliably detect, and coding assumptions about what to do in those conditions is a much dicier prospect.
A very simple answer I’ve found for the question as it is, is to run:
pip -V # (Uppercase V)