Using a conda env created on windows for linux
Question:
I am in the process of migrating a number of environments from PC to linux.
On windows I run:
C:foo> conda env export > environment.yml
And, later, on linux, I run:
$ conda env create -f environment.yml
But I get errors like:
ResolvePackageNotFound:
- icc_rt=2017.0.4
- vc=14
- vs2015_runtime=14.0.25123
- wincertstore=0.2
- qt==5.9.5=vc14he4a7d60_0
This must (I assume) be a solved problem.
May someone smarter than I on this topic please let me know how they would go about this?
Thanks!
Answers:
Yes and no. Using conda export will enable someone else to exactly replicate your environment. This implicitly assumes you are on the same platform.
Unfortunately, when swapping platforms, you need to handle packages that are platform dependent. The easiest way is just remove them. Keep in mind that if you include a high level package with lots of dependencies, all of those dependencies are looked up/handled by conda.
For example, if you want to include pandas, you don’t need to include numpy, qt, matplotlib, and dateutils in your environment spec. Just listing pandas is enough, conda takes care of the rest.
In this way, you may be better off just listing out the bare minimum of your environment requirements by hand in a text editor.
Alternatively, you can use conda export, but you may still need to remove a good number of the build numbers (i.e. =vc17gnad8qt6h) and packages that are Windows only (like wincertstore).
When exporting your environment, use the option –from-history. It will export just the libs you explicitly installed, and not the dependencies:
conda env export --from-history > environment.yml
No platform specific info will be exported. It will prevent a lot of headaches.
Usually some dependencies are platform specific. Also the default conda env export put platform specific info in the libs. This will make an environment.yml file created in Windows fail when trying to recreate it in Linux, and vice-versa.
Conda and Pip aren’t really very good to perfectly recreate an environment in another machine, since they don’t record all dependencies of dependencies. Usually you won’t have trouble, but if you have, it will be hard to spot that a dependency of a dependency is a slightly different version.
Extra tip: always install a lib referencing its version number (e.g.: conda install pandas=1.2.1). Without the version, the command above will export the dependencies without a version, ruining the reproducibility of your environment.
But what if you created your env from a environment.yml file? Now --from-history will export the platform specific dependencies. Then grep is your friend. You will need to grep all of your import statements, see which of them are defined in your environment.yml file and just use them using the same version without platform specific info. Better to start doing it correctly using the –from-history or manually editing the file.
Extra extra tip: Python and Conda default tooling still aren’t very good to create reproducible installs between platforms. Nowadays I’ve been successfully using Poetry for it.
I am in the process of migrating a number of environments from PC to linux.
On windows I run:
C:foo> conda env export > environment.yml
And, later, on linux, I run:
$ conda env create -f environment.yml
But I get errors like:
ResolvePackageNotFound:
- icc_rt=2017.0.4
- vc=14
- vs2015_runtime=14.0.25123
- wincertstore=0.2
- qt==5.9.5=vc14he4a7d60_0
This must (I assume) be a solved problem.
May someone smarter than I on this topic please let me know how they would go about this?
Thanks!
Yes and no. Using conda export will enable someone else to exactly replicate your environment. This implicitly assumes you are on the same platform.
Unfortunately, when swapping platforms, you need to handle packages that are platform dependent. The easiest way is just remove them. Keep in mind that if you include a high level package with lots of dependencies, all of those dependencies are looked up/handled by conda.
For example, if you want to include pandas, you don’t need to include numpy, qt, matplotlib, and dateutils in your environment spec. Just listing pandas is enough, conda takes care of the rest.
In this way, you may be better off just listing out the bare minimum of your environment requirements by hand in a text editor.
Alternatively, you can use conda export, but you may still need to remove a good number of the build numbers (i.e. =vc17gnad8qt6h) and packages that are Windows only (like wincertstore).
When exporting your environment, use the option –from-history. It will export just the libs you explicitly installed, and not the dependencies:
conda env export --from-history > environment.yml
No platform specific info will be exported. It will prevent a lot of headaches.
Usually some dependencies are platform specific. Also the default conda env export put platform specific info in the libs. This will make an environment.yml file created in Windows fail when trying to recreate it in Linux, and vice-versa.
Conda and Pip aren’t really very good to perfectly recreate an environment in another machine, since they don’t record all dependencies of dependencies. Usually you won’t have trouble, but if you have, it will be hard to spot that a dependency of a dependency is a slightly different version.
Extra tip: always install a lib referencing its version number (e.g.: conda install pandas=1.2.1). Without the version, the command above will export the dependencies without a version, ruining the reproducibility of your environment.
But what if you created your env from a environment.yml file? Now --from-history will export the platform specific dependencies. Then grep is your friend. You will need to grep all of your import statements, see which of them are defined in your environment.yml file and just use them using the same version without platform specific info. Better to start doing it correctly using the –from-history or manually editing the file.
Extra extra tip: Python and Conda default tooling still aren’t very good to create reproducible installs between platforms. Nowadays I’ve been successfully using Poetry for it.