Why do people create virtualenv in a docker container?
Question:
You can build a container with Dockerfile in a few seconds.
Then why do people need to install a virtual enviroment inside the docker container?
It’s like a "virtual machine" in a virtual machine ?
Answers:
I am working with virtualenvs in Docker and I think there are several reasons:
- you may want to isolate your app from system’s python packages
- you may want to run a custom version of python but still keep the system’s packages untouched
- you may need fine grain control on the packages installed for a specific app
- you may need to run multiple apps with different requirements
I think these are all reasonably good reasons to add a little pip install virtualenv at the end of the installation! 🙂
Here is my two cents, or rather comments on @gru ‘s answer and some of the comments.
- Neither docker nor virtual environments are virtual machines
- every line in your docker file produces overhead. But it’s true that at runtime virtual environments have zero impact
- the idea of docker containers is that you have one process which interacts with other (docker-)services in a client-server relationship. Running different apps in one docker or calling one app from another inside a docker is somehow against that idea. More importantly, it adds complexity to you docker, which you want to avoid.
- “isolating” the python packages that the app sees (inside a virtual environment) from the packages installed in the docker is only necessary if you need to assure a certain version for one or more packages.
- the system installed inside the container only serves as an environment for the one app that you are running. Adjust it to the requirements of your app. There is no need to leave it “untouched”
So in conclusion: There is no good reason for using a virtual environment inside a container. Install whatever packages you need on the system. If you need control over the exact package versions, install the (docker-wide) with pip or alike.
If you think that you need to run different apps with different package versions inside a single container, take a step back and rethink your design. You are heading towards more complexity, more difficult maintenance and more headache. Split the work/services up into several containers.
In our team, we avoid creating Python virtual environments inside Docker containers.
It adds yet another layer of complexity that is pretty much useless in most cases.
In very rare cases, it might be required to not alter system tools that you need and that are written in Python and require some specific versions of Python libraries that would otherwise be overwritten by your own Python package manager at build time.
People might also do it because they are used to virtual environments and keep using them inside Docker images without asking themselves too much questions: it still works, and it might be the default setup they find in tutorials.
Now, these tutorials might suggest creating a virtual environment inside Docker images to be on the safe side, and to avoid all possible clashes with system’s python tools.
For instance, Poetry is a package manager, but it also insists on using or creating virtual environments for you. I believe that this is not a proper separation of concerns: it would be better to have one tool for package management, and another tool for virtual environment management, if needed.
But because Poetry is mixing up these things (in an understandable attempt to have one-tool-that-solves-it-all), we had to iterate several times on our Dockerfile before we found a simple workaround:
FROM python:3.10-slim
RUN pip install -U pip poetry
# creates working directory
RUN mkdir -p /app/src
WORKDIR /app/src
# installs dependencies
ADD poetry.lock pyproject.toml /app/src/
RUN poetry export -f requirements.txt --output requirements.txt
RUN pip install --no-deps -r requirements.txt
# copies the rest of the source code
ADD . /app/src/
# …
No virtualenv needed, and builds now seem to be faster when dependencies are simply installed with pip.
You can build a container with Dockerfile in a few seconds.
Then why do people need to install a virtual enviroment inside the docker container?
It’s like a "virtual machine" in a virtual machine ?
I am working with virtualenvs in Docker and I think there are several reasons:
- you may want to isolate your app from system’s python packages
- you may want to run a custom version of python but still keep the system’s packages untouched
- you may need fine grain control on the packages installed for a specific app
- you may need to run multiple apps with different requirements
I think these are all reasonably good reasons to add a little pip install virtualenv at the end of the installation! 🙂
Here is my two cents, or rather comments on @gru ‘s answer and some of the comments.
- Neither docker nor virtual environments are virtual machines
- every line in your docker file produces overhead. But it’s true that at runtime virtual environments have zero impact
- the idea of docker containers is that you have one process which interacts with other (docker-)services in a client-server relationship. Running different apps in one docker or calling one app from another inside a docker is somehow against that idea. More importantly, it adds complexity to you docker, which you want to avoid.
- “isolating” the python packages that the app sees (inside a virtual environment) from the packages installed in the docker is only necessary if you need to assure a certain version for one or more packages.
- the system installed inside the container only serves as an environment for the one app that you are running. Adjust it to the requirements of your app. There is no need to leave it “untouched”
So in conclusion: There is no good reason for using a virtual environment inside a container. Install whatever packages you need on the system. If you need control over the exact package versions, install the (docker-wide) with pip or alike.
If you think that you need to run different apps with different package versions inside a single container, take a step back and rethink your design. You are heading towards more complexity, more difficult maintenance and more headache. Split the work/services up into several containers.
In our team, we avoid creating Python virtual environments inside Docker containers.
It adds yet another layer of complexity that is pretty much useless in most cases.
In very rare cases, it might be required to not alter system tools that you need and that are written in Python and require some specific versions of Python libraries that would otherwise be overwritten by your own Python package manager at build time.
People might also do it because they are used to virtual environments and keep using them inside Docker images without asking themselves too much questions: it still works, and it might be the default setup they find in tutorials.
Now, these tutorials might suggest creating a virtual environment inside Docker images to be on the safe side, and to avoid all possible clashes with system’s python tools.
For instance, Poetry is a package manager, but it also insists on using or creating virtual environments for you. I believe that this is not a proper separation of concerns: it would be better to have one tool for package management, and another tool for virtual environment management, if needed.
But because Poetry is mixing up these things (in an understandable attempt to have one-tool-that-solves-it-all), we had to iterate several times on our Dockerfile before we found a simple workaround:
FROM python:3.10-slim
RUN pip install -U pip poetry
# creates working directory
RUN mkdir -p /app/src
WORKDIR /app/src
# installs dependencies
ADD poetry.lock pyproject.toml /app/src/
RUN poetry export -f requirements.txt --output requirements.txt
RUN pip install --no-deps -r requirements.txt
# copies the rest of the source code
ADD . /app/src/
# …
No virtualenv needed, and builds now seem to be faster when dependencies are simply installed with pip.