How does virtualenv work?
Question:
I checked the activate script and it looks to me all it does is:
- set VIRTUAL_ENV env
- append $VIRTUAL_ENV/bin in front of PATH
How does virtualenv provide that magical virtual environment by these? What do I miss?
Answers:
I will describe the basic process, which I learned from the presentation @jcollado linked to.
When Python starts, it looks at the path of the binary, and the prefixes thereof.
So let’s say your virtualenv is /home/blah/scratch. The Python process knows it was executed from /home/blah/scratch/bin/python (which is usually just a copy of your system python binary /usr/bin/python) and it knows its own version X.Y because it’s compiled into it. Then Python looks for lib/pythonX.Y/os.py in this order:
/home/blah/scratch/bin/lib/pythonX.Y/os.py
/home/blah/scratch/lib/pythonX.Y/os.py <-- this file should exist
/home/blah/lib/pythonX.Y/os.py
/home/lib/pythonX.Y/os.py
/lib/pythonX.Y/os.py
It stops at /home/blah/scratch/lib/pythonX.Y/os.py because it’s the first file that actually exists. If it didn’t, Python would keep looking. It then sets sys.prefix based on this. It uses a similar process to set sys.exec_prefix, and then sys.path is constructed based on these.
- First the user creates a new virtualenv with the command
virtualenv myenv. This creates a directory called myenv and copies the system python binary to myenv/bin. It also adds other necessary files and directories to myenv, including a setup script in bin/activate and a lib subdirectory for modules and packages.
- Then the user sources the activate script with
. myenv/bin/activate, which sets the shell’s PATH environment variable to start with myenv/bin.
- Now when the user runs
python from this shell, it will execute the copy of the binary stored in myenv/bin. Even though the binary is identical to the one in /usr/bin/python, the standard python binary is designed to search for packages and modules in directories that are relative to the binary’s path (this functionality is not related to virtualenv). It looks in ../lib/pythonX.Y where X and Y are the major and minor version numbers of the python binary. So now it is looking in myenv/lib/pythonX.Y.
- The myenv/bin directory also contains a script named
pip so that when the user installs new packages using pip from the virtualenv, they will be installed in myenv/lib/pythonX.Y
I checked the activate script and it looks to me all it does is:
- set VIRTUAL_ENV env
- append $VIRTUAL_ENV/bin in front of PATH
How does virtualenv provide that magical virtual environment by these? What do I miss?
I will describe the basic process, which I learned from the presentation @jcollado linked to.
When Python starts, it looks at the path of the binary, and the prefixes thereof.
So let’s say your virtualenv is /home/blah/scratch. The Python process knows it was executed from /home/blah/scratch/bin/python (which is usually just a copy of your system python binary /usr/bin/python) and it knows its own version X.Y because it’s compiled into it. Then Python looks for lib/pythonX.Y/os.py in this order:
/home/blah/scratch/bin/lib/pythonX.Y/os.py
/home/blah/scratch/lib/pythonX.Y/os.py <-- this file should exist
/home/blah/lib/pythonX.Y/os.py
/home/lib/pythonX.Y/os.py
/lib/pythonX.Y/os.py
It stops at /home/blah/scratch/lib/pythonX.Y/os.py because it’s the first file that actually exists. If it didn’t, Python would keep looking. It then sets sys.prefix based on this. It uses a similar process to set sys.exec_prefix, and then sys.path is constructed based on these.
- First the user creates a new virtualenv with the command
virtualenv myenv. This creates a directory called myenv and copies the system python binary to myenv/bin. It also adds other necessary files and directories to myenv, including a setup script in bin/activate and a lib subdirectory for modules and packages. - Then the user sources the activate script with
. myenv/bin/activate, which sets the shell’sPATHenvironment variable to start with myenv/bin. - Now when the user runs
pythonfrom this shell, it will execute the copy of the binary stored in myenv/bin. Even though the binary is identical to the one in /usr/bin/python, the standard python binary is designed to search for packages and modules in directories that are relative to the binary’s path (this functionality is not related to virtualenv). It looks in ../lib/pythonX.Y where X and Y are the major and minor version numbers of the python binary. So now it is looking in myenv/lib/pythonX.Y. - The myenv/bin directory also contains a script named
pipso that when the user installs new packages using pip from the virtualenv, they will be installed in myenv/lib/pythonX.Y