Testing - KosinskiLab/AlphaPulldown GitHub Wiki

Testing

This page describes how AlphaPulldown tests are organized, what runs in CI, where to find coverage reports, and which checks still require explicit cluster or manual validation before a release.

Test suite layout

AlphaPulldown uses pytest for active CPU-side testing. The test tree is split by purpose:

  • test/unit/: fast unit tests for pure helpers and small mocked components.
  • test/integration/: CPU-only integration tests for filesystem, CLI, and module wiring.
  • test/functional/: deterministic package-level tests that are heavier than unit/integration.
  • test/cluster/: Slurm, GPU, and workstation smoke tests, wrappers, and cluster-only utilities.
  • test/alphalink/: AlphaLink fixtures and optional-test support files used by AlphaLink helper tests and smoke runs.
  • test/outdated/: legacy tests kept only as a migration area; they are not part of default collection.
  • test/test_data/: shared fixtures used by the active suites.

See also:

  • test/README.md for the maintained test layout
  • test/RELEASE_READINESS.md for the practical release gate and the difference between always-on CI coverage and explicit cluster/manual validation

Pytest markers

The active pytest.ini markers are:

  • unit: fast unit tests
  • integration: CPU-only integration tests
  • functional: heavier but deterministic package tests
  • cluster: tests intended for cluster execution
  • gpu: tests that require a GPU
  • slow: intentionally slow tests
  • external_tools: tests that need optional heavyweight tools
  • network: tests that need network access

By default, local pytest runs exclude:

not cluster and not gpu and not network and not external_tools

This keeps the default suite CPU-only and CI-friendly.

Recommended local commands

Install the test dependencies:

pip install -e .[test]

Run the default active CPU suite:

pytest

Run only unit tests in parallel:

pytest -n auto --dist loadfile test/unit

Run the CPU CI-equivalent suite:

pytest -n auto --dist loadfile test/unit
pytest test/integration

Run coverage locally:

pytest -n auto --dist loadfile test/unit \
  --cov=alphapulldown \
  --cov-config=.coveragerc \
  --cov-report=

pytest test/integration \
  --cov=alphapulldown \
  --cov-config=.coveragerc \
  --cov-append \
  --cov-report=

coverage report --skip-covered --show-missing
coverage xml -o coverage.xml
coverage json -o coverage.json
coverage html -d htmlcov
python test/tools/check_function_coverage.py --report-only coverage.json

Run cluster or GPU smoke wrappers explicitly on a suitable environment:

python test/cluster/run_alphafold2_predictions.py
python test/cluster/run_alphafold3_predictions.py

Some MMseqs-backed functional or cluster checks are intentionally opt-in:

export RUN_MMSEQS_FUNCTIONAL_TESTS=1

Coverage policy

Coverage is scoped to the alphapulldown/ package via .coveragerc.

  • Vendored or nested projects such as alphafold, alphafold3, ColabFold, and AlphaLink2 are out of scope.
  • alphapulldown/analysis_pipeline/af2plots is omitted from package coverage.

There are two coverage views:

  1. Standard line and branch coverage from pytest-cov
  2. A separate AST-based function coverage report from test/tools/check_function_coverage.py

The function coverage script is currently report-only. It helps identify functions that are still never executed by the active CPU suites, but should be treated as an audit signal rather than proof of workflow correctness.

GitHub Actions

The main GitHub Actions workflow runs the following jobs:

1. Smoke tests

Runs on Python 3.10 and 3.11:

pytest -n auto --dist loadfile test/unit
pytest test/integration

These jobs are CPU-only and do not collect coverage.

2. Coverage

Runs on Python 3.11:

  • unit coverage pass
  • integration coverage pass with --cov-append
  • coverage report
  • coverage xml
  • coverage json
  • coverage html
  • AST function coverage report

The coverage job enforces the package coverage floor configured in the workflow.

3. Container builds

The workflow also builds AlphaFold2 and AlphaFold3 containers.

These are useful packaging checks, but they are not substitutes for explicit cluster/GPU inference smoke tests.

Where to see coverage

Locally

After a coverage run:

  • terminal summary:

    coverage report --skip-covered --show-missing

  • XML report: coverage.xml

  • JSON report: coverage.json

  • HTML report: htmlcov/index.html

In GitHub Actions

Open the coverage job and download the coverage-reports artifact. It contains:

  • coverage.json
  • coverage.xml
  • htmlcov/

On pushes to main, the HTML coverage report is also published via GitHub Pages.

GPU and cluster tests

GPU inference and Slurm wrapper tests are intentionally not part of default CI because GitHub-hosted runners do not provide GPUs or Slurm.

These tests live under test/cluster/ and must be run explicitly on an appropriate cluster environment.

Examples of release-critical checks that require cluster/manual validation:

  • AF3 wrapper output isolation for combined JSON folds
  • MMseqs-generated .a3m and .pkl / .pkl.xz feature artifacts
  • AF3 species-pairing regressions (e.g. issue #588)
  • AF2/AF3 dimer inference sanity (e.g. ipTM > 0.6)

Current testing strategy

The current direction is to favor:

  • direct unit tests for simple helpers
  • CPU-only integration tests for wiring and module interaction
  • explicit GPU/cluster smoke tests only where real inference is needed

This keeps CI reasonably fast while still improving real coverage of alphapulldown/.

Compatibility note

The package metadata currently advertises Python ≥3.8, but GitHub Actions exercises Python 3.10 and 3.11.

If Python 3.8 remains a supported target, it should be treated as a compatibility commitment beyond the current CI matrix.