How to share conda environments across platforms
Question:
The conda docs at http://conda.pydata.org/docs/using/envs.html explain how to share environments with other people.
However, the docs tell us this is not cross platform:
NOTE: These explicit spec files are not usually cross platform, and
therefore have a comment at the top such as # platform: osx-64 showing the
platform where they were created. This platform is the one where this spec
file is known to work. On other platforms, the packages specified might not
be available or dependencies might be missing for some of the key packages
already in the spec.
NOTE: Conda does not check architecture or dependencies when installing
from an explicit specification file. To ensure the packages work correctly,
be sure that the file was created from a working environment and that it is
used on the same architecture, operating system and platform, such as linux-
64 or osx-64.
Is there a good method to share and recreate a conda environment in one platform (e.g. CentOS) in another platform (e.g. Windows)?
Answers:
This answer is given with the assumption that you would like to make sure that
the same versions of the packages that you generally care about are on
different platforms and that you don’t care about the exact same versions of
all packages in the entire dependency tree. If you are trying to install the
exact same version of all packages in your entire dependency tree that has a
high likelihood of failure since some conda packages have different
dependencies for osx/win/linux. For example, the recipe for
otrobopt
will install different packages on Win vs. osx/linux, so the environment list
would be different.
Recommendation: manually create an environment.yaml file and specify or pin
only the dependencies that you care about. Let the conda solver do the rest.
Probably worth noting is that conda-env (the tool that you use to manage conda
environments) explicitly recommends that you "Always create your
environment.yml file by hand."
Then you would just do conda env create --file environment.yml
Have a look at the readme for
conda-env.
They can be quite simple:
name: basic_analysis
dependencies:
- numpy
- pandas
Or more complex where you pin dependencies and specify anaconda.org channels to
install from:
name: stats-web
channels:
- javascript
dependencies:
- python=3.4 # or 2.7 if you are feeling nostalgic
- bokeh=0.9.2
- numpy=1.9
- nodejs=0.10
- flask
- pip:
- Flask-Testing
In either case, you can create an environment with conda env create --file environment.yaml.
NOTE: You may need to use .* as a version suffix if you’re using an older version of conda.
conda-env export should be used used to export your complete environment to file named my_env.yml.
Check the working solution on getting only prefix on OS X instead of complete dependency including pip.
Step 1: deactivate from the environment if activated. else it will create yml file with only prefix.
Step 2: run below command to export
conda-env export -n my_env > my_env.yml
it will export every required dependency, channel and pip install in a yml file which is importable to share with others.
Step 3: run below command to import
conda-env create -n my_env -f= my_env.yml
it will create the exact environment as is on sharing fellow machine.
For those interested in a solution to maintain a single environment file that can be used in Linux, macOS, and Windows, please check the conda-devenv tool at https://github.com/ESSS/conda-devenv.
Whilst it is possible to create your environment.yml file by hand, you can ensure that your environment works across platforms by using the conda env export --from-history flag.
This will only include packages that you’ve explicitly asked for, as opposed to including every package in your environment.
For example, if you create an environment and install a package conda install python=3.8 numpy, it will install numerous other dependencies as well as python and numpy.
If you then run conda env export > environment.yml, your environment.yml file will include all the additional dependencies conda automatically installed for you.
On the other hand, running conda env export --from-history will just create environment.yml with python=3.8 and numpy and thus will work across platforms.
Answer adapted from the docs.
An aspect missing from the other answers is that the question asker mentions "spec files" and not "environment.yml" file. These are different.
Spec file
A spec file specifies the exact package URL’s and is used to recreate identical environments (on the same platform).
It looks like this:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
@EXPLICIT
https://repo.anaconda.com/pkgs/free/osx-64/mkl-11.3.3-0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/numpy-1.11.1-py35_0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/openssl-1.0.2h-1.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/pip-8.1.2-py35_0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/python-3.5.2-0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/readline-6.2-2.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/setuptools-25.1.6-py35_0.tar.bz2
It can be obtained with conda list --explicit from the conda environment of interest.
To create a new environment with it one would use the conda create command:
conda create --name <env_name> --file <spec file path>
environment.yml
The environment.yml file is described well in this answer.
It can be obtained with the following commands from the conda environment of interest:
conda env export to get all packages in the current environmentconda env export --from-history to only get the packages explicitly installed (i.e. not automatically added depenedencies)
This question is quite old and conda has developed in the meantime. Perhaps the original meaning of spec file was equal to environment.yml files, but for completeness I am adding this answer.
None of these solutions worked out for me, the problem the OP has raised is about platform dependent suffixes added in the dependency making it impossible to be used it in a cross-platform way.
Turns out the solution for this is to export the environment with an additional option called no-builds
Suppose you want to export your environment from MacOS to Debian.
1.) Invoke conda env export --no-builds > env_macos.yml
2.) Invoke cp env_macos.yml env_debian.yml
3.) Move env_debian.yml to your Debian host
4.) conda env create -f env_debian.yml
Soon after you do 4, again there may be same package resolving related issues for certain packages, just remove those entries alone and invoke 4 again. Things will work.
The conda docs at http://conda.pydata.org/docs/using/envs.html explain how to share environments with other people.
However, the docs tell us this is not cross platform:
NOTE: These explicit spec files are not usually cross platform, and
therefore have a comment at the top such as # platform: osx-64 showing the
platform where they were created. This platform is the one where this spec
file is known to work. On other platforms, the packages specified might not
be available or dependencies might be missing for some of the key packages
already in the spec.
NOTE: Conda does not check architecture or dependencies when installing
from an explicit specification file. To ensure the packages work correctly,
be sure that the file was created from a working environment and that it is
used on the same architecture, operating system and platform, such as linux-
64 or osx-64.
Is there a good method to share and recreate a conda environment in one platform (e.g. CentOS) in another platform (e.g. Windows)?
This answer is given with the assumption that you would like to make sure that
the same versions of the packages that you generally care about are on
different platforms and that you don’t care about the exact same versions of
all packages in the entire dependency tree. If you are trying to install the
exact same version of all packages in your entire dependency tree that has a
high likelihood of failure since some conda packages have different
dependencies for osx/win/linux. For example, the recipe for
otrobopt
will install different packages on Win vs. osx/linux, so the environment list
would be different.
Recommendation: manually create an environment.yaml file and specify or pin
only the dependencies that you care about. Let the conda solver do the rest.
Probably worth noting is that conda-env (the tool that you use to manage conda
environments) explicitly recommends that you "Always create your
environment.yml file by hand."
Then you would just do conda env create --file environment.yml
Have a look at the readme for
conda-env.
They can be quite simple:
name: basic_analysis
dependencies:
- numpy
- pandas
Or more complex where you pin dependencies and specify anaconda.org channels to
install from:
name: stats-web
channels:
- javascript
dependencies:
- python=3.4 # or 2.7 if you are feeling nostalgic
- bokeh=0.9.2
- numpy=1.9
- nodejs=0.10
- flask
- pip:
- Flask-Testing
In either case, you can create an environment with conda env create --file environment.yaml.
NOTE: You may need to use .* as a version suffix if you’re using an older version of conda.
conda-env export should be used used to export your complete environment to file named my_env.yml.
Check the working solution on getting only prefix on OS X instead of complete dependency including pip.
Step 1: deactivate from the environment if activated. else it will create yml file with only prefix.
Step 2: run below command to export
conda-env export -n my_env > my_env.yml
it will export every required dependency, channel and pip install in a yml file which is importable to share with others.
Step 3: run below command to import
conda-env create -n my_env -f= my_env.yml
it will create the exact environment as is on sharing fellow machine.
For those interested in a solution to maintain a single environment file that can be used in Linux, macOS, and Windows, please check the conda-devenv tool at https://github.com/ESSS/conda-devenv.
Whilst it is possible to create your environment.yml file by hand, you can ensure that your environment works across platforms by using the conda env export --from-history flag.
This will only include packages that you’ve explicitly asked for, as opposed to including every package in your environment.
For example, if you create an environment and install a package conda install python=3.8 numpy, it will install numerous other dependencies as well as python and numpy.
If you then run conda env export > environment.yml, your environment.yml file will include all the additional dependencies conda automatically installed for you.
On the other hand, running conda env export --from-history will just create environment.yml with python=3.8 and numpy and thus will work across platforms.
Answer adapted from the docs.
An aspect missing from the other answers is that the question asker mentions "spec files" and not "environment.yml" file. These are different.
Spec file
A spec file specifies the exact package URL’s and is used to recreate identical environments (on the same platform).
It looks like this:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
@EXPLICIT
https://repo.anaconda.com/pkgs/free/osx-64/mkl-11.3.3-0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/numpy-1.11.1-py35_0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/openssl-1.0.2h-1.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/pip-8.1.2-py35_0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/python-3.5.2-0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/readline-6.2-2.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/setuptools-25.1.6-py35_0.tar.bz2
It can be obtained with conda list --explicit from the conda environment of interest.
To create a new environment with it one would use the conda create command:
conda create --name <env_name> --file <spec file path>
environment.yml
The environment.yml file is described well in this answer.
It can be obtained with the following commands from the conda environment of interest:
conda env exportto get all packages in the current environmentconda env export --from-historyto only get the packages explicitly installed (i.e. not automatically added depenedencies)
This question is quite old and conda has developed in the meantime. Perhaps the original meaning of spec file was equal to environment.yml files, but for completeness I am adding this answer.
None of these solutions worked out for me, the problem the OP has raised is about platform dependent suffixes added in the dependency making it impossible to be used it in a cross-platform way.
Turns out the solution for this is to export the environment with an additional option called no-builds
Suppose you want to export your environment from MacOS to Debian.
1.) Invoke conda env export --no-builds > env_macos.yml
2.) Invoke cp env_macos.yml env_debian.yml
3.) Move env_debian.yml to your Debian host
4.) conda env create -f env_debian.yml
Soon after you do 4, again there may be same package resolving related issues for certain packages, just remove those entries alone and invoke 4 again. Things will work.