Install python package on a specific folder instead of user level
Question:
So I am currently working on a DevOps task for a Python repository.
I am tasked with making the repository work properly with our Docker environment.
My current issue is that when running pip3 install, it installs everything in various folders that have nothing to do with the current repository path.
This is not ideal when working with docker-compose for a local development environment.
What we want to achieve is to install pip dependencies once on the host machine and then mount them as a dynamic volume inside the docker container. The reason is simple, we want to be able to destroy the container as many times as we want without having to download everything again. Because this is an AI repository, you can imagine that the pip dependencies are heavy and take a lot of time to download. We currently have about 5 to 10GB of dependencies to download depending on the branch and current code progress.
Ideally, I’d like pip to behave like Node.JS tools such as yarn or npm where it builds a local folder (like node_modules) that I can easily mount inside the container.
I have looked through the pip documentation but I simply can’t find a way to do this at all.
Is there a way to constrain pip and python to install dependencies inside a specific folder and to only use those dependencies?
Answers:
Have you tried using virtual environments:
- Setup a virtual environment in your persistent mounted shared folder.
- Install all dependencies in your virtual environment.
- Mount the drive to your container.
- Activate the virtual environment
- Now, when you use python commands, they will be from the virutal environment.
ps. simplist way to setup virtual environment is using (https://docs.python.org/3/library/venv.html) but there are other solutions to do that too.
Docker Approach
You can just split up your dependencies into a separate docker image which serves as the basis for your current image. This way you only need to build base-image once and then only when the dependencies change.
Base Image Example
FROM python:3.10-bullseye
# Copy requirements into container and install
COPY ./requirements.txt .
pip install -r requirements.txt
Build with docker build -t myImage .
Actual Image
FROM myImage
# setup for everything else
Python Approach
In general pip provides the --target option to install the packages into a specific directory instead of the site-packages folder see documentation.
However, the use of a virtual environment is recommended to separate the dependencies from the host into an environment, which can then be mounted into a docker-container.
# create venv once
python -m venv /path/myVenv
# enter venv
source /path/myVenv/bin/activate
# install dependencies
pip install -r requirements.txt
Then you can just mount /path/myVenv into the container and source again to access it.
So I am currently working on a DevOps task for a Python repository.
I am tasked with making the repository work properly with our Docker environment.
My current issue is that when running pip3 install, it installs everything in various folders that have nothing to do with the current repository path.
This is not ideal when working with docker-compose for a local development environment.
What we want to achieve is to install pip dependencies once on the host machine and then mount them as a dynamic volume inside the docker container. The reason is simple, we want to be able to destroy the container as many times as we want without having to download everything again. Because this is an AI repository, you can imagine that the pip dependencies are heavy and take a lot of time to download. We currently have about 5 to 10GB of dependencies to download depending on the branch and current code progress.
Ideally, I’d like pip to behave like Node.JS tools such as yarn or npm where it builds a local folder (like node_modules) that I can easily mount inside the container.
I have looked through the pip documentation but I simply can’t find a way to do this at all.
Is there a way to constrain pip and python to install dependencies inside a specific folder and to only use those dependencies?
Have you tried using virtual environments:
- Setup a virtual environment in your persistent mounted shared folder.
- Install all dependencies in your virtual environment.
- Mount the drive to your container.
- Activate the virtual environment
- Now, when you use python commands, they will be from the virutal environment.
ps. simplist way to setup virtual environment is using (https://docs.python.org/3/library/venv.html) but there are other solutions to do that too.
Docker Approach
You can just split up your dependencies into a separate docker image which serves as the basis for your current image. This way you only need to build base-image once and then only when the dependencies change.
Base Image Example
FROM python:3.10-bullseye
# Copy requirements into container and install
COPY ./requirements.txt .
pip install -r requirements.txt
Build with docker build -t myImage .
Actual Image
FROM myImage
# setup for everything else
Python Approach
In general pip provides the --target option to install the packages into a specific directory instead of the site-packages folder see documentation.
However, the use of a virtual environment is recommended to separate the dependencies from the host into an environment, which can then be mounted into a docker-container.
# create venv once
python -m venv /path/myVenv
# enter venv
source /path/myVenv/bin/activate
# install dependencies
pip install -r requirements.txt
Then you can just mount /path/myVenv into the container and source again to access it.