Good practices for managing optional dependencies - GlacioHack/xdem GitHub Wiki

Here's a list of good practices for managing optional dependencies:

Package definition

This is the obvious one.

One needs to define optional dependencies in setup.cfg or pyproject.toml in "extras".

Such as:

[options.extras_require]
opt =
    laspy[lazrs]
    scikit-image
    numba
    dask
    matplotlib
    psutil
    plotly

Source code

The best is usually to catch and re-raise custom ImportError within each optional function through a helper.

It is usually better to use try/except to try to import myoptionaldep within each function rather than at the top of a file (i.e. near other imports), for three reasons: (i) To have access to the upstream import error directly when raising the new custom error (with a helpful message on how to install the missing dependency), (ii) To avoid setting a global variable _has_myoptionaldep and (iii) to only trigger the optional import on runtime. Writing a helper function can facilitate catching and raising an error consistently across varying optional dependencies of the package.

For instance:

def import_optional(import_name: str, package_name: str | None = None, extra_name: str = "opt") -> Any:
    """
    Helper function to consistently import and raise errors for an optional dependency.

    :param import_name: Name of the dependency to import.
    :param package_name: Name of the package to install (optional, only if different from import name e.g.
        "pyyaml" package is imported as "import yaml".).
    :param extra_name: Name of the extra tag to install the optional dependency.
    """

    if package_name is None:
        package_name = import_name

    try:
        return __import__(import_name)
    except ImportError as e:
        raise ImportError(
            f"Optional dependency '{package_name}' required. "
            f"Install it directly or through: pip install geoutils[{extra_name}]."
        ) from e

which is called within a function requiring an optional dependency the following way:

def myoptionalfunc()
    myoptionaldep = import_optional("myoptionaldep")

    # Rest of the code that uses myoptionaldep...

Exception: An exception to this are functions that NEED to be defined before runtime, such as decorated functions, to-be-compiled Numba functions (that don't accept inside import) or inherited classes (that need to be defined during MyClass(OptionalClass). In this case, use a try/except at head of file, and add a "fake" decorator or object subclass for when the optional dependency does not exist.

try:
    from numba import jit, prange
except:
    jit = # Fake decorator
    prange = # Fake prange

@jit
def myfunc_compiled():
    for i in prange():
        ...

def myfunc_calling_compiled:
     # Raise import error ahead, in a separate function calling the Numba code
     import_optional("numba")
     myfunc_compiled()

Tests

Multiple environments in CI with an import skip for functional tests + a missing dep importerror verification.

To ensure all scenarios are tested thoroughly, the best is to:

• Run CI with several environments: only required, then required + optional dependencies,
• Add pytest.importorskip("myoptionaldep") in all tests (functions or whole module) that require an optional dependency,
• Add a test checking the ImportError is raised properly for each function relying on an optional dependency that should, which should be skipped when the package exists, for example using pytest.mark.skip(importlib.util.find_spec("myoptionaldep") is not None, reason="Only runs if myoptionaldep is missing")

def test_myoptionalfunc()
   """This test only runs if myoptionaldep is INSTALLED."""

   pytest.importorskip("myoptionaldep")

   myoptionalfunc()

pytest.mark.skip(importlib.util.find_spec("myoptionaldep") is not None, reason="Only runs if myoptionaldep is missing")
def test_myoptionalfunc__missing_dep():
   """This test only runs if myoptionaldep is MISSING."""

    with pytest.raises(ImportError, match="Optional dependency 'myoptionaldep' required.*):
        myoptionalfunc()

Type checking

Sometimes, we need to use type hinting of an optional dependency, for instance assuming matplotlib is an optional dependency:

def myplotfunc(colormap: matplotlib.colors.Colormap, other_args: Any)
   
     matplotlib = import_optional("matplotlib")      

     # Code using matplotlib...

To make this code work, there exists a recommended solution using typing:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    import matplotlib

# Previous code unchanged...

This allows to import matplotlib only for the purpose of type checking, making the inside-function import work the same as above.