What's Python good practice for importing and offering optional features?

The try: method does not need to be global — it can be used in any scope and so modules can be “lazy-loaded” at runtime. For example:

def foo():
    try:
        import external_module
    except ImportError:
        external_module = None 

    if external_module:
        external_module.some_whizzy_feature()
    else:
        print("You could be using a whizzy feature right now, if you had external_module.")

When your script is run, no attempt will be made to load external_module. The first time foo() is called, external_module is (if available) loaded and inserted into the function’s local scope. Subsequent calls to foo() reinsert external_module into its scope without needing to reload the module.

In general, it’s best to let Python handle import logic — it’s been doing it for a while. 🙂

One way to handle the problem of different dependencies for different features is to implement the optional features as plugins. That way the user has control over which features are activated in the app but isn’t responsible for tracking down the dependencies herself. That task then gets handled at the time of each plugin’s installation.

Not sure if this is good practice, but I created a function that does the optional import (using importlib) and error handling:

def _optional_import(module: str, name: str = None, package: str = None):
    import importlib
    try:
        module = importlib.import_module(module)
        return module if name is None else getattr(module, name)
    except ImportError as e:
        if package is None:
            package = module
        msg = f"install the '{package}' package to make use of this feature"
        raise ValueError(msg) from e

If an optional module is not available, the user will at least get the idea what to do. E.g.

# code ...

if file.endswith('.json'):
    from json import load
elif file.endswith('.yaml'):
    # equivalent to 'from yaml import safe_load as load'
    load = _optional_import('yaml', 'safe_load', package='pyyaml')

# code using load ...

The main disadvantage with this approach is that your imports have to be done in-line and are not all on the top of your file. Therefore, it might be considered better practice to use a slight adaptation of this function (assuming that you are importing a function or the like):

def _optional_import_(module: str, name: str = None, package: str = None):
    import importlib
    try:
        module = importlib.import_module(module)
        return module if name is None else getattr(module, name)
    except ImportError as e:
        if package is None:
            package = module
        msg = f"install the '{package}' package to make use of this feature"
        import_error = e

        def _failed_import(*args):
            raise ValueError(msg) from import_error

        return _failed_import

Now, you can make the imports with the rest of your imports and the error will only be raised when the function that failed to import is actually used. E.g.

from utils import _optional_import_  # let's assume we import the function
from json import load as json_load
yaml_load = _optional_import_('yaml', 'safe_load', package='pyyaml')

# unimportant code ...

with open('test.txt', 'r') as fp:
    result = yaml_load(fp)    # will raise a value error if import was not successful

PS: sorry for the late answer!

I’m really excited to share this new technique I came up with to handle optional dependencies!

The concept is to produce the error when the uninstalled package is used not imported.

Just add a single call before your imports. You don’t need to change any code at all. No more using try: when importing. No more using conditional skip decorators when writing tests.

Main components

• An importer to return a fake module for missing imports
• A fake module that raises an exception when it’s used
• A custom Exception that will skip tests automatically if raised within one

Minimal Code Example

import sys
import importlib
from unittest.case import SkipTest
from _pytest.outcomes import Skipped

class MissingOptionalDependency(SkipTest, Skipped):
    def __init__(self, msg=None):
        self.msg = msg
    def __repr__(self):
        return f"MissingOptionalDependency: {self.msg}" if self.msg else f"MissingOptionalDependency"

class GeneralImporter:
    def __init__(self, *names):
        self.names = names
        sys.meta_path.insert(0, self)
    def find_spec(self, fullname, path=None, target=None):
        if fullname in self.names:
            return importlib.util.spec_from_loader(fullname, self)
    def create_module(self, spec):
        return FakeModule(name=spec.name)
    def exec_module(self, module):
        pass

class FakeModule:
    def __init__(self, name):
        self.name = name
    def __call__(self, *args, **kwargs):
        raise MissingOptionalDependency(f"Optional dependency '{self.name}' was used but it isn't installed.")

GeneralImporter("notinstalled")
import notinstalled  # No error
print(notinstalled)  # <__main__.FakeModule object at 0x0000014B7F6D9E80>
notinstalled()  # MissingOptionalDependency: Optional dependency 'notinstalled' was used but it isn't installed.

Package

The technique above has some shortcomings that my package fixes.

It’s open-source, lightweight, and has no dependencies!

Some key differences to the example above:

• Covers more than 100 dunder methods (All tested)
• Covers 15 common dunder attribute lookups
• Entry function is generalimport which returns an ImportCatcher
  • ImportCatcher holds names, scope, and caught names
    • It can be enabled and disabled
    • The scope prevents external packages from being affected
• Wildcard support to allow any package to be imported
• Puts the importer first in sys.meta_path
  • Lets it catch namespace imports (Usually occurs with uninstalled packages)

Generalimport on GitHub

pip install generalimport

Minimal example

from generalimport import generalimport
generalimport("notinstalled")

from notinstalled import missing_func  # No error
missing_func()  # Error occurs here

The readme on GitHub goes more in-depth

Here’s a production-grade solution, using importlib and Pandas’s import_optional_dependency as suggested by @dre-hh

from typing import *
import importlib, types

def module_exists(
        *names: Union[List[str], str],
        error: str = "ignore",
        warn_every_time: bool = False,
        __INSTALLED_OPTIONAL_MODULES: Dict[str, bool] = {}
) -> Optional[Union[Tuple[types.ModuleType, ...], types.ModuleType]]:
    """
    Try to import optional dependencies.
    Ref: https://stackoverflow.com/a/73838546/4900327

    Parameters
    ----------
    names: str or list of strings.
        The module name(s) to import.
    error: str {'raise', 'warn', 'ignore'}
        What to do when a dependency is not found.
        * raise : Raise an ImportError.
        * warn: print a warning.
        * ignore: If any module is not installed, return None, otherwise,
          return the module(s).
    warn_every_time: bool
        Whether to warn every time an import is tried. Only applies when error="warn".
        Setting this to True will result in multiple warnings if you try to
        import the same library multiple times.
    Returns
    -------
    maybe_module : Optional[ModuleType, Tuple[ModuleType...]]
        The imported module(s), if all are found.
        None is returned if any module is not found and `error!="raise"`.
    """
    assert error in {"raise", "warn", "ignore"}
    if isinstance(names, (list, tuple, set)):
        names: List[str] = list(names)
    else:
        assert isinstance(names, str)
        names: List[str] = [names]
    modules = []
    for name in names:
        try:
            module = importlib.import_module(name)
            modules.append(module)
            __INSTALLED_OPTIONAL_MODULES[name] = True
        except ImportError:
            modules.append(None)

    def error_msg(missing: Union[str, List[str]]):
        if not isinstance(missing, (list, tuple)):
            missing = [missing]
        missing_str: str = ' '.join([f'"{name}"' for name in missing])
        dep_str = 'dependencies'
        if len(missing) == 1:
            dep_str = 'dependency'
        msg = f'Missing optional {dep_str} {missing_str}. Use pip or conda to install.'
        return msg

    missing_modules: List[str] = [name for name, module in zip(names, modules) if module is None]
    if len(missing_modules) > 0:
        if error == "raise":
            raise ImportError(error_msg(missing_modules))
        if error == "warn":
            for name in missing_modules:
                ## Ensures warning is printed only once
                if warn_every_time is True or name not in __INSTALLED_OPTIONAL_MODULES:
                    print(f'Warning: {error_msg(name)}')
                    __INSTALLED_OPTIONAL_MODULES[name] = False
        return None
    if len(modules) == 1:
        return modules[0]
    return tuple(modules)

Usage: ignore errors (error="ignore", default behavior)

Suppose we want to run certain code only if the required libraries exists:

if module_exists("pydantic", "sklearn"):
    from pydantic import BaseModel
    from sklearn.metrics import accuracy_score
    class AccuracyCalculator(BaseModel):
        num_decimals: int = 5
        def calculate(self, y_pred: List, y_true: List) -> float:
            return round(accuracy_score(y_true, y_pred), self.num_decimals)
    print("Defined AccuracyCalculator in global context")

If either dependencies pydantic or skelarn do not exist, then the class AccuracyCalculator will not be defined and the print statement will not run.

Usage: raise ImportError (error="raise")

Alternatively, you can raise a error if any module does not exist:

if module_exists("pydantic", "sklearn", error="raise"):
    from pydantic import BaseModel
    from sklearn.metrics import accuracy_score
    class AccuracyCalculator(BaseModel):
        num_decimals: int = 5
        def calculate(self, y_pred: List, y_true: List) -> float:
            return round(accuracy_score(y_true, y_pred), self.num_decimals)
    print("Defined AccuracyCalculator in global context")

Output:

line 60, in module_exists(error, __INSTALLED_OPTIONAL_MODULES, *names)
     58 if len(missing_modules) > 0:
     59     if error == "raise":
---> 60         raise ImportError(error_msg(missing_modules))
     61     if error == "warn":
     62         for name in missing_modules:

ImportError: Missing optional dependencies "pydantic" "sklearn". Use pip or conda to install.

Usage: print a warning (error="warn")

Alternatively, you can print a warning if the module does not exist.

if module_exists("pydantic", "sklearn", error="warn"):
    from pydantic import BaseModel
    from sklearn.metrics import accuracy_score
    class AccuracyCalculator(BaseModel):
        num_decimals: int = 5
        def calculate(self, y_pred: List, y_true: List) -> float:
            return round(accuracy_score(y_true, y_pred), self.num_decimals)
    print("Defined AccuracyCalculator in global context")
    
if module_exists("pydantic", "sklearn", error="warn"):
    from pydantic import BaseModel
    from sklearn.metrics import roc_auc_score
    class RocAucCalculator(BaseModel):
        num_decimals: int = 5
        def calculate(self, y_pred: List, y_true: List) -> float:
            return round(roc_auc_score(y_true, y_pred), self.num_decimals)
    print("Defined RocAucCalculator in global context")

Output:

Warning: Missing optional dependency "pydantic". Use pip or conda to install.
Warning: Missing optional dependency "sklearn". Use pip or conda to install.

Here, we ensure that only one warning is printed for each missing module, otherwise you would get a warning each time you try to import.

This is very useful for Python libraries where you might try to import the same optional dependencies many times, and only want to see one Warning.

You can pass warn_every_time=True to always print the warning when you try to import.

Another option is to use @contextmanager and with. In this situation, you do not know beforehand which dependencies are needed:

from contextlib import contextmanager

@contextmanager
def optional_dependencies(error: str = "ignore"):
    assert error in {"raise", "warn", "ignore"}
    try:
        yield None
    except ImportError as e:
        if error == "raise":
            raise e
        if error == "warn":
            msg = f'Missing optional dependency "{e.name}". Use pip or conda to install.'
            print(f'Warning: {msg}')

Usage:

with optional_dependencies("warn"):
    import module_which_does_not_exist_1
    import module_which_does_not_exist_2
    z = 1
print(z)

Output:

Warning: Missing optional dependency "module_which_does_not_exist_1". Use pip or conda to install.


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In [43], line 5
      3     import module_which_does_not_exist_2
      4     z = 1
----> 5 print(z)

NameError: name 'z' is not defined

Here, you should define all your imports immediately after with. The first module which is not installed will throw ImportError, which is caught by optional_dependencies. Depending on how you want to handle this error, it will either ignore it, print a warning, or raise it again.

The entire code will only run if all the modules are installed.