Keeping the same, shared virtualenvs when switching from pyenv-virtualenv to pipenv
Question:
I started looking at pipenv and it seems to be pretty good. My only concern is, that most of my projects involve numpy, scipy and some other not-so-small libraries.
The current way manage my projects:
I have pyenv and pyenv-virtualenv installed. I have a few (currently 4) specific virtualenvs that each cater to a type of project. The projects themselves have .pyenv-version set, I have the autoload virtualenv feature of pyenv enabled. If I need to share a project, I generate a requirements.txt with pip freeze -l from the virtualenv.
So in my current setup, I have X number of projects and Y, Y << X number of virtualenvs, all amounting to a few GB of harddisk space. Note that because of large libraries like numpy each of the virtualenvs are pretty big, around 700-900 MB.
My question:
As far as I understand, pipenv will, by default create a virtualenv for all of my projects, so the harddisk space taken up by my virtualenvs would increase considerably. What I’m interested in is:
- is it possible to share
pipenv environments across several projects, that use exactly the same dependencies? i.e. multiple pipenv configs that load the same virtualenv?
- if not, is it possible to generate
pipenv config files from a virtualenv I set up with pyenv? i.e. I would not use pipenv to actually run my projects, I would not create any virtualenvs with pipenv, but I would create pipenv config files for sharing the project (in this case, probably along side a requirements.txt as well).
edit:
I rewrote most of the question to make it clearer.
Answers:
pipenv doesn’t seem to be a good fit for your specific workflow because it’s project-centric rather than environment-centric. pipenv treats a virtual environment as volatile and reserves the right to alter it freely if circumstances call for it. You can use it but in the case of alterations to your environments, it will be a pain to keep all projects synchronized due to pipenv‘s stricter scrunity.
You can explicitly specify a virtual environment for pipenv to use for a project by creating a .venv file in the project root with a path to it (normally, virtualenvs are created in a specific location with autogenerated names that include a hash of the path to the project). This seems to be undocumented.
However, pipenv, unlike virtualenv, checks and enforces that the virtual environment has the exact set of modules satisfying conditions in Pipfile and the exact "last tested configuration" specified in a generated Pipfile.lock.
- Besides, if you generate
Pipfile from requirements.txt, it will specify exact package versions and include all dependencies — while by design, it’s supposed to contain more intelligent information.
So, if you change any package version in an environment, you’ll need to:
- update all
Pipfile.locks in affected projects (e.g. copy the changed one). With a generated Pipfile, you may get away with deleting them instead.
- update all
Pipfiles in affected projects to the new package versions (e.g. copy the changed one) if there was a change
I started looking at pipenv and it seems to be pretty good. My only concern is, that most of my projects involve numpy, scipy and some other not-so-small libraries.
The current way manage my projects:
I have pyenv and pyenv-virtualenv installed. I have a few (currently 4) specific virtualenvs that each cater to a type of project. The projects themselves have .pyenv-version set, I have the autoload virtualenv feature of pyenv enabled. If I need to share a project, I generate a requirements.txt with pip freeze -l from the virtualenv.
So in my current setup, I have X number of projects and Y, Y << X number of virtualenvs, all amounting to a few GB of harddisk space. Note that because of large libraries like numpy each of the virtualenvs are pretty big, around 700-900 MB.
My question:
As far as I understand, pipenv will, by default create a virtualenv for all of my projects, so the harddisk space taken up by my virtualenvs would increase considerably. What I’m interested in is:
- is it possible to share
pipenvenvironments across several projects, that use exactly the same dependencies? i.e. multiplepipenvconfigs that load the samevirtualenv? - if not, is it possible to generate
pipenvconfig files from avirtualenvI set up withpyenv? i.e. I would not usepipenvto actually run my projects, I would not create anyvirtualenvswithpipenv, but I would createpipenvconfig files for sharing the project (in this case, probably along side arequirements.txtas well).
edit:
I rewrote most of the question to make it clearer.
pipenv doesn’t seem to be a good fit for your specific workflow because it’s project-centric rather than environment-centric. pipenv treats a virtual environment as volatile and reserves the right to alter it freely if circumstances call for it. You can use it but in the case of alterations to your environments, it will be a pain to keep all projects synchronized due to pipenv‘s stricter scrunity.
You can explicitly specify a virtual environment for pipenv to use for a project by creating a .venv file in the project root with a path to it (normally, virtualenvs are created in a specific location with autogenerated names that include a hash of the path to the project). This seems to be undocumented.
However, pipenv, unlike virtualenv, checks and enforces that the virtual environment has the exact set of modules satisfying conditions in Pipfile and the exact "last tested configuration" specified in a generated Pipfile.lock.
- Besides, if you generate
Pipfilefromrequirements.txt, it will specify exact package versions and include all dependencies — while by design, it’s supposed to contain more intelligent information.
So, if you change any package version in an environment, you’ll need to:
- update all
Pipfile.locks in affected projects (e.g. copy the changed one). With a generatedPipfile, you may get away with deleting them instead. - update all
Pipfiles in affected projects to the new package versions (e.g. copy the changed one) if there was a change