conda: what difference does it make if we set pip_interop_enabled=True?

Question:

There are many posts on this site which reference, typically in passing, the idea of setting pip_interop_enabled=True within some environment. This makes conda and pip3 somehow interact better, I am told. To be precise, people say conda will search PyPI for packages that don’t exist in the main channels if this is true. They also say it’s "experimental."

Here is conda’s documentation about this. It notes that much of conda’s behavior in recent versions has also improved even with pip_interop_enabled=False, leading to questions about what this setting even does.

Here is my question: in real terms, what does all of this mean?

  • Is the only difference that conda will search PyPI if this is True and not if it’s False?
  • Are there other things that it does? For instance, if I need to install some package from pip, will conda know better not to clobber it if this setting is True?
  • What, to be precise, goes wrong if I set this to True? Are there known edge cases that somehow break things if this "experimental" setting is set to True?
  • Why would I ever not want to set this?
Asked By: Mike Battaglia

||

Answers:

Not a PyPI Searching Feature

First, let’s clarify: Conda will not "search PyPI" – that is not what the pip_interop_enabled configuration option adds. Rather, it enables the solver to allow a package already installed with pip to satisfy a dependency requirement of a Conda package. Note that the option is about Pip interoperability (as distinct from PyPI) and it doesn’t matter whether the package was sourced from PyPI, GitHub, local, etc..

Example: scipy -> numpy

Let’s consider a simple example to illustrate the behavior. Start with the following environment that has Python 3.10 and numpy installed from PyPI.

pip_interop.yaml

name: pip_interop
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip

  ## PyPI packages
  - pip:
    - numpy

which we can create with

conda env create -n pip_interop -f pip_interop.yaml

and verify that the numpy is from PyPI:

$ conda list -n pip_interop numpy
# packages in environment at /Users/user/mambaforge/envs/pip_interop:
#
# Name                    Version                   Build  Channel
numpy                     1.24.2                   pypi_0  

Let’s see what would happen installing scipy and in particular, how it satisfies its numpy dependency.

Installing without Pip interoperability

In default mode, we see the following behavior

$ conda install -n pip_interop scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    numpy-1.24.2               |  py310h788a5b3_0         6.1 MB
    scipy-1.10.0               |  py310h240c617_2        20.2 MB
    ------------------------------------------------------------
                                           Total:        27.4 MB

The following NEW packages will be INSTALLED:

  appdirs            conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0 
  brotlipy           conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 
  certifi            conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  cffi               conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  cryptography       conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 
  libblas            conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas 
  libcblas           conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas 
  libcxx             conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 
  libgfortran5       conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 
  liblapack          conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas 
  libopenblas        conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 
  llvm-openmp        conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 
  numpy              conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0 
  packaging          conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 
  pooch              conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0 
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 
  pyopenssl          conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 
  python_abi         conda-forge/osx-64::python_abi-3.10-3_cp310 
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 
  scipy              conda-forge/osx-64::scipy-1.10.0-py310h240c617_2 
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 

Proceed ([y]/n)?

Observe that despite numpy already being installed in the environment, Conda is proposing to replace it with a Conda version. That is, only considers the information in conda-meta/ to determine whether a package is installed and won’t check the environment’s lib/python3.10/site-packages/.

Installing with Pip interoperability

Now we try it with the pip_interop_enabled turned on:

$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n foo scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    scipy-1.10.0               |  py310h240c617_2        20.2 MB
    ------------------------------------------------------------
                                           Total:        21.3 MB

The following NEW packages will be INSTALLED:

  appdirs            conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0 
  brotlipy           conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 
  certifi            conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  cffi               conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  cryptography       conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 
  libblas            conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas 
  libcblas           conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas 
  libcxx             conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 
  libgfortran5       conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 
  liblapack          conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas 
  libopenblas        conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 
  llvm-openmp        conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 
  packaging          conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 
  pooch              conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0 
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 
  pyopenssl          conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 
  python_abi         conda-forge/osx-64::python_abi-3.10-3_cp310 
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 
  scipy              conda-forge/osx-64::scipy-1.10.0-py310h240c617_2 
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 


Proceed ([y]/n)?

Note that now the numpy is not proposed to be replaced and this is because the existing pip-installed version is consider able to satisfy the dependency.

Why is this experimental?

There may be multiple reasons why this remains experimental after several years. One important reason is that Conda only tests its package builds against Conda builds of the dependencies. So, it cannot guarantee that the packages are functionally exchangeable.

Furthermore, Conda packages often bring in non-Python dependencies. There has been a rise in wheel deployments, which is the PyPI approach to this, but isn’t ubiquitous. There are still many "wrapper" packages out there where the PyPI version assumes some binary is on PATH, whereas the installation of the Conda package guarantees the binary is also installed.

Another important issue is that the PyPI-Conda name mapping is not well-defined. That is, the name of a package in PyPI may not correspond to its Conda package name. This can directly lead to cryptic issues when the names diverge. Specifically, Conda will not correctly recognize that a pip-installed package satisfies the requirement when the names don’t match. Hence, the is some unexpected heterogeneity in how the interoperability applies.

Example: torch vs pytorch

In the Python ecosystem, the torch module is provided by the PyPI package torch. However, the package torch in PyPI goes by pytorch on Conda channels.

Here’s how this can lead to inconsistent behavior. Let’s begin with torch installed from PyPI:

pip_interop.yaml

name: pip_interop
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip

  ## PyPI packages
  - pip:
    - torch

Creating with:

conda env create -n pip_interop -f pip_interop.yaml

Now if we install torchvision from Conda, even with the pip_interop_enabled on, we get:

$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n pip_interop torchvision
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    jpeg-9e                    |       hb7f2c08_3         226 KB
    libprotobuf-3.21.12        |       hbc0c0cd_0         1.8 MB
    mkl-2022.2.1               |   h44ed08c_16952       113.1 MB
    numpy-1.24.2               |  py310h788a5b3_0         6.1 MB
    pillow-9.4.0               |  py310h306a057_1        44.1 MB
    pytorch-1.13.1             |cpu_py310h2bbf33f_1        56.9 MB
    sleef-3.5.1                |       h6db0672_2         1.0 MB
    torchvision-0.14.1         |cpu_py310hd5ee960_0         5.9 MB
    ------------------------------------------------------------
                                           Total:       230.1 MB

The following NEW packages will be INSTALLED:

  brotlipy           conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 
  certifi            conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  cffi               conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  cryptography       conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 
  freetype           conda-forge/osx-64::freetype-2.12.1-h3f81eb7_1 
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 
  jpeg               conda-forge/osx-64::jpeg-9e-hb7f2c08_3 
  lcms2              conda-forge/osx-64::lcms2-2.14-h29502cd_1 
  lerc               conda-forge/osx-64::lerc-4.0.0-hb486fe8_0 
  libblas            conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas 
  libcblas           conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas 
  libcxx             conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 
  libdeflate         conda-forge/osx-64::libdeflate-1.17-hac1461d_0 
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 
  libgfortran5       conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 
  liblapack          conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas 
  libopenblas        conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 
  libpng             conda-forge/osx-64::libpng-1.6.39-ha978bb4_0 
  libprotobuf        conda-forge/osx-64::libprotobuf-3.21.12-hbc0c0cd_0 
  libtiff            conda-forge/osx-64::libtiff-4.5.0-hee9004a_2 
  libwebp-base       conda-forge/osx-64::libwebp-base-1.2.4-h775f41a_0 
  libxcb             conda-forge/osx-64::libxcb-1.13-h0d85af4_1004 
  llvm-openmp        conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 
  mkl                conda-forge/osx-64::mkl-2022.2.1-h44ed08c_16952 
  numpy              conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0 
  openjpeg           conda-forge/osx-64::openjpeg-2.5.0-h13ac156_2 
  pillow             conda-forge/osx-64::pillow-9.4.0-py310h306a057_1 
  pthread-stubs      conda-forge/osx-64::pthread-stubs-0.4-hc929b4f_1001 
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 
  pyopenssl          conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 
  python_abi         conda-forge/osx-64::python_abi-3.10-3_cp310 
  pytorch            conda-forge/osx-64::pytorch-1.13.1-cpu_py310h2bbf33f_1 
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 
  sleef              conda-forge/osx-64::sleef-3.5.1-h6db0672_2 
  tbb                conda-forge/osx-64::tbb-2021.7.0-hb8565cd_1 
  torchvision        conda-forge/osx-64::torchvision-0.14.1-cpu_py310hd5ee960_0 
  typing_extensions  conda-forge/noarch::typing_extensions-4.4.0-pyha770c72_0 
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 
  xorg-libxau        conda-forge/osx-64::xorg-libxau-1.0.9-h35c211d_0 
  xorg-libxdmcp      conda-forge/osx-64::xorg-libxdmcp-1.1.3-h35c211d_0 
  zstd               conda-forge/osx-64::zstd-1.5.2-hbc0c0cd_6 


Proceed ([y]/n)?

That is, Conda still tries to install pytorch and this means that it will lead to clobbering of the existing torch package installed from PyPI. This has the potential to having residual files from the clobbered version of the package intermixed with the clobbering version.

Basically, this is undefined behavior and the Conda software may not give you any warning about potential problems.

Answered By: merv
Categories: questions Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.