Compare packages between two anaconda installations
Question:
I had two versions of Python, with Anaconda, installed in my Mac, 3.5 and 3.7. Is there any way to know what packages I had installed in 3.5 that are not in 3.7, like those that you can install with pip (pulp, wordcloud, and alike).
What I have tried so far is using command line with this:
diff -rq anaconda/.../python3.5/site-packages anaconda3/.../python3.7/site-packages
This shows the differences between the directories but shows a lot of information of repeated packages and core modules. How can I find the differences in packages between two Anaconda versions?
Answers:
You can list all the packages installed under a specific environment using conda list. Also, you can export the list to a file with the flag --export.
For example, if your environments are named python3 and pytorch_p36, you can do the following:
conda list -n python3 --export > python3-packages.txt
conda list -n pytorch_p36 --export > pytorch_p36-packages.txt
diff python3-packages.txt pytorch_p36-packages.txt
The command also accepts the flag --json, which will export the data in a json format, suitable for automatic processing.
Thanks dataista for your solution for Linux. For Windows diff does not work, but in Windows one can use FC:
conda list -n python3 --export > python3-packages.txt
conda list -n pytorch_p36 --export > pytorch_p36-packages.txt
FC python3-packages.txt pytorch_p36-packages.txt
A recent development now provides the ability to compare a conda environment against a requirements file. The only mention of this I’ve found so far is in a pull request that was merged back on July 16, 2020. I can verify that the compare command works in conda version 4.8.4. It should be noted that only differences are listed in the output.
Usage:
(ProcMCD43A1) C:Usersjustincase>conda compare environment.yml --json
[
"argon2-cffi found but mismatch. Specification pkg: argon2-cffi==20.1.0=py37he774522_1, Running pkg: argon2-cffi==20.1.0=py37h4ab8f01_1",
"attrs found but mismatch. Specification pkg: attrs==19.3.0=py_0, Running pkg: attrs==20.2.0=pyh9f0ad1d_0",
"backcall found but mismatch. Specification pkg: backcall==0.2.0=py_0, Running pkg: backcall==0.2.0=pyh9f0ad1d_0",
"bleach found but mismatch. Specification pkg: bleach==3.1.5=py_0, Running pkg: bleach==3.2.0=pyh9f0ad1d_0",
"brotlipy found but mismatch. Specification pkg: brotlipy==0.7.0=py37he774522_1000, Running pkg: brotlipy==0.7.0=py37h4ab8f01_1000",
"bzip2 found but mismatch. Specification pkg: bzip2==1.0.8=he774522_0, Running pkg: bzip2==1.0.8=he774522_3",
"cffi found but mismatch. Specification pkg: cffi==1.14.0=py37h7a1dbc1_0, Running pkg: cffi==1.14.3=py37h26f1ce3_0",
"cfitsio found but mismatch. Specification pkg: cfitsio==3.470=he774522_5, Running pkg: cfitsio==3.470=hbbe6aef_6",
"cftime found but mismatch. Specification pkg: cftime==1.2.1=py37h2a96729_0, Running pkg: cftime==1.2.1=py37h44b1f71_0",
"chardet found but mismatch. Specification pkg: chardet==3.0.4=py37_1003, Running pkg: chardet==3.0.4=py37hc8dfbb8_1006",
"click not found",
"click-plugins not found",
"cligj not found",
"cryptography found but mismatch. Specification pkg: cryptography==2.9.2=py37h7a1dbc1_0, Running pkg: cryptography==3.1=py37h26f1ce3_0",
"curl found but mismatch. Specification pkg: curl==7.67.0=h2a8f88b_0, Running pkg: curl==7.71.1=h4b64cdc_5",
"cycler found but mismatch. Specification pkg: cycler==0.10.0=py37_0, Running pkg: cycler==0.10.0=py_2",
"descartes not found",
"entrypoints found but mismatch. Specification pkg: entrypoints==0.3=py37_0, Running pkg: entrypoints==0.3=py37hc8dfbb8_1001",
"fiona not found",
"freexl found but mismatch. Specification pkg: freexl==1.0.5=hfa6e2cd_0, Running pkg: freexl==1.0.5=hd288d7e_1002",
"gdal found but mismatch. Specification pkg: gdal==3.0.2=py37hdf43c64_0, Running pkg: gdal==3.1.2=py37h6ddc196_1",
"geopandas not found",
"geos found but mismatch. Specification pkg: geos==3.8.0=h33f27b4_0, Running pkg: geos==3.8.1=he025d50_0",
"geotiff found but mismatch. Specification pkg: geotiff==1.5.1=h5770a2b_1, Running pkg: geotiff==1.6.0=h09e6dc1_1",
"h5py not found",
"hdf4 found but mismatch. Specification pkg: hdf4==4.2.13=h712560f_2, Running pkg: hdf4==4.2.13=hf8e6fe8_1003",
"hdf5 found but mismatch. Specification pkg: hdf5==1.10.4=h7ebc959_0, Running pkg: hdf5==1.10.6=nompi_he0bbb20_101",
...
]
Using mini conda 4.11, with the new compare command, I’m going to compare a named environment with a path environment.
❯ conda --version
conda 4.11.0
First list the environments I have
❯ conda env list
# conda environments:
#
base * /Users/me/miniforge3
experiments /Users/me/miniforge3/envs/experiments
tfm1 /Users/me/miniforge3/envs/tfm1
/Users/me/path/project/env
Now I want to compare the conda env in the directory ./path/project/env with the named env tfm1 (my tensorflow env)…
First I switch to the tfm1 env and export its package list:
❯ conda activate tfm1
❯ conda env export -f /Users/me/path/tfm1_env.yml
Next I switch to the other env, in the directory:
❯ conda activate /Users/me/path/project/env
And finally use the new compare command:
❯ conda compare /Users/me/path/tfm1_env.yml
black not found
certifi found but mismatch. Specification pkg: certifi==2021.10.8=py38h10201cd_1, Running pkg: certifi==2021.10.8=pypi_0
click not found
dataclasses not found
executing found but mismatch. Specification pkg: executing==0.8.2=pyhd8ed1ab_0, Running pkg: executing==0.8.3=pyhd8ed1ab_0
flit-core found but mismatch. Specification pkg: flit-core==3.6.0=pyhd8ed1ab_0, Running pkg: flit-core==3.7.1=pyhd8ed1ab_0
fonttools found but mismatch. Specification pkg: fonttools==4.29.1=py38hea4295b_0, Running pkg: fonttools==4.31.2=py38h33210d7_0
fribidi not found
...
The documentation is not too clear on this, but "Specification pkg" refers to a package found in the environment whose specification we exported to a file ("tfm1" in my example). "Running pkg" is, of course, the one I activated.
Even less clear, the "not found" refers to the active environment: the packages "black", "fribidi", etc, are in my tfm1 (specified) but not in my project/env (active) environment.
Also look at the pyenvdiff package at pyenvdiff
Rather ironically, pyenvdiff does not yet have a conda package. Anyone care to create and submit a conda-forge recipe for pyenvdiff?
I had two versions of Python, with Anaconda, installed in my Mac, 3.5 and 3.7. Is there any way to know what packages I had installed in 3.5 that are not in 3.7, like those that you can install with pip (pulp, wordcloud, and alike).
What I have tried so far is using command line with this:
diff -rq anaconda/.../python3.5/site-packages anaconda3/.../python3.7/site-packages
This shows the differences between the directories but shows a lot of information of repeated packages and core modules. How can I find the differences in packages between two Anaconda versions?
You can list all the packages installed under a specific environment using conda list. Also, you can export the list to a file with the flag --export.
For example, if your environments are named python3 and pytorch_p36, you can do the following:
conda list -n python3 --export > python3-packages.txt
conda list -n pytorch_p36 --export > pytorch_p36-packages.txt
diff python3-packages.txt pytorch_p36-packages.txt
The command also accepts the flag --json, which will export the data in a json format, suitable for automatic processing.
Thanks dataista for your solution for Linux. For Windows diff does not work, but in Windows one can use FC:
conda list -n python3 --export > python3-packages.txt
conda list -n pytorch_p36 --export > pytorch_p36-packages.txt
FC python3-packages.txt pytorch_p36-packages.txt
A recent development now provides the ability to compare a conda environment against a requirements file. The only mention of this I’ve found so far is in a pull request that was merged back on July 16, 2020. I can verify that the compare command works in conda version 4.8.4. It should be noted that only differences are listed in the output.
Usage:
(ProcMCD43A1) C:Usersjustincase>conda compare environment.yml --json
[
"argon2-cffi found but mismatch. Specification pkg: argon2-cffi==20.1.0=py37he774522_1, Running pkg: argon2-cffi==20.1.0=py37h4ab8f01_1",
"attrs found but mismatch. Specification pkg: attrs==19.3.0=py_0, Running pkg: attrs==20.2.0=pyh9f0ad1d_0",
"backcall found but mismatch. Specification pkg: backcall==0.2.0=py_0, Running pkg: backcall==0.2.0=pyh9f0ad1d_0",
"bleach found but mismatch. Specification pkg: bleach==3.1.5=py_0, Running pkg: bleach==3.2.0=pyh9f0ad1d_0",
"brotlipy found but mismatch. Specification pkg: brotlipy==0.7.0=py37he774522_1000, Running pkg: brotlipy==0.7.0=py37h4ab8f01_1000",
"bzip2 found but mismatch. Specification pkg: bzip2==1.0.8=he774522_0, Running pkg: bzip2==1.0.8=he774522_3",
"cffi found but mismatch. Specification pkg: cffi==1.14.0=py37h7a1dbc1_0, Running pkg: cffi==1.14.3=py37h26f1ce3_0",
"cfitsio found but mismatch. Specification pkg: cfitsio==3.470=he774522_5, Running pkg: cfitsio==3.470=hbbe6aef_6",
"cftime found but mismatch. Specification pkg: cftime==1.2.1=py37h2a96729_0, Running pkg: cftime==1.2.1=py37h44b1f71_0",
"chardet found but mismatch. Specification pkg: chardet==3.0.4=py37_1003, Running pkg: chardet==3.0.4=py37hc8dfbb8_1006",
"click not found",
"click-plugins not found",
"cligj not found",
"cryptography found but mismatch. Specification pkg: cryptography==2.9.2=py37h7a1dbc1_0, Running pkg: cryptography==3.1=py37h26f1ce3_0",
"curl found but mismatch. Specification pkg: curl==7.67.0=h2a8f88b_0, Running pkg: curl==7.71.1=h4b64cdc_5",
"cycler found but mismatch. Specification pkg: cycler==0.10.0=py37_0, Running pkg: cycler==0.10.0=py_2",
"descartes not found",
"entrypoints found but mismatch. Specification pkg: entrypoints==0.3=py37_0, Running pkg: entrypoints==0.3=py37hc8dfbb8_1001",
"fiona not found",
"freexl found but mismatch. Specification pkg: freexl==1.0.5=hfa6e2cd_0, Running pkg: freexl==1.0.5=hd288d7e_1002",
"gdal found but mismatch. Specification pkg: gdal==3.0.2=py37hdf43c64_0, Running pkg: gdal==3.1.2=py37h6ddc196_1",
"geopandas not found",
"geos found but mismatch. Specification pkg: geos==3.8.0=h33f27b4_0, Running pkg: geos==3.8.1=he025d50_0",
"geotiff found but mismatch. Specification pkg: geotiff==1.5.1=h5770a2b_1, Running pkg: geotiff==1.6.0=h09e6dc1_1",
"h5py not found",
"hdf4 found but mismatch. Specification pkg: hdf4==4.2.13=h712560f_2, Running pkg: hdf4==4.2.13=hf8e6fe8_1003",
"hdf5 found but mismatch. Specification pkg: hdf5==1.10.4=h7ebc959_0, Running pkg: hdf5==1.10.6=nompi_he0bbb20_101",
...
]
Using mini conda 4.11, with the new compare command, I’m going to compare a named environment with a path environment.
❯ conda --version
conda 4.11.0
First list the environments I have
❯ conda env list
# conda environments:
#
base * /Users/me/miniforge3
experiments /Users/me/miniforge3/envs/experiments
tfm1 /Users/me/miniforge3/envs/tfm1
/Users/me/path/project/env
Now I want to compare the conda env in the directory ./path/project/env with the named env tfm1 (my tensorflow env)…
First I switch to the tfm1 env and export its package list:
❯ conda activate tfm1
❯ conda env export -f /Users/me/path/tfm1_env.yml
Next I switch to the other env, in the directory:
❯ conda activate /Users/me/path/project/env
And finally use the new compare command:
❯ conda compare /Users/me/path/tfm1_env.yml
black not found
certifi found but mismatch. Specification pkg: certifi==2021.10.8=py38h10201cd_1, Running pkg: certifi==2021.10.8=pypi_0
click not found
dataclasses not found
executing found but mismatch. Specification pkg: executing==0.8.2=pyhd8ed1ab_0, Running pkg: executing==0.8.3=pyhd8ed1ab_0
flit-core found but mismatch. Specification pkg: flit-core==3.6.0=pyhd8ed1ab_0, Running pkg: flit-core==3.7.1=pyhd8ed1ab_0
fonttools found but mismatch. Specification pkg: fonttools==4.29.1=py38hea4295b_0, Running pkg: fonttools==4.31.2=py38h33210d7_0
fribidi not found
...
The documentation is not too clear on this, but "Specification pkg" refers to a package found in the environment whose specification we exported to a file ("tfm1" in my example). "Running pkg" is, of course, the one I activated.
Even less clear, the "not found" refers to the active environment: the packages "black", "fribidi", etc, are in my tfm1 (specified) but not in my project/env (active) environment.
Also look at the pyenvdiff package at pyenvdiff
Rather ironically, pyenvdiff does not yet have a conda package. Anyone care to create and submit a conda-forge recipe for pyenvdiff?