Things to know on profiling and benchmarking - GlacioHack/xdem GitHub Wiki

GeoUtils separates profiling, large data tests and benchmarks. Below is what each piece means and where it lives.

Profiler

The profiler is in geoutils/profiler.py, and aims to be reusable through the public API.

Its main helper, geoutils.profiler.profile_call(), runs one function and returns the result plus runtime and peak RAM. It uses psutil for the main Python process. If a distributed Dask client is active, it also uses Dask's MemorySampler to report worker RAM and spilled memory. This is the measurement layer reused by tests and benchmarks.

Benchmark workflows

A benchmark workflow is a reusable code path that we want to measure (CPU/RAM) depending on raster/pointcloud size.

The workflows are defined in benchmarks/_dask_workflows.py. They create lazy rasters larger than worker memory, run package operations we want to test such as reproject() followed by mean(), and finish with a small result: a scalar, a sample, or a chunked write. They are shared by the heavy pytest tests and ASV.

Large data tests

The large data tests are pytest tests in benchmarks/test_large_data.py. They run the workflows on a constrained Dask LocalCluster and check that the result is correct and execution stays out-of-core. They are marked as large_data, skipped by default and only run when pytest is called with -large-data. They are triggered in CI during every PR commit, they run in parallel, and use a single Python version and OS to reduce computational costs.

ASV benchmarks

Airspeed Velocity (ASV) is a benchmarking tool used to record timings and memory metrics across commits.

The ASV configuration is in asv.conf.json, and the benchmark classes are in benchmarks/. They reuse the workflows defined above and expose time_* methods for runtime and track_* methods for memory metrics. ASV can also compare implemented strategies, for example the GeoUtils polygonize strategies label_union, label_stitch and geometry_stitch.

CI and publishing

Normal PR CI runs the large data tests (large-data-tests.yml) and a quick ASV check only (benchmarks-asv-check.yml) that uses the environment variable GEOUTILS_ASV_PR_CHECK=1 to simplify the checks.

The separate GitHub Actions workflow benchmarks-asv.yml runs scheduled (currently every two weeks) or manually triggered full ASV benchmark, then uploads the results to an asv-results branch on the package. Once results are uploaded to this branch, it triggers the workflow benchmark-publish.yml that retrieves the results and deploys the GitHub webpage.

Commands

For running ASV locally:

asv check -E existing
asv run --quick --show-stderr -E existing

For running the large dataset locally, possibly with tuned parameters through environment variables (first two lines optional):

GEOUTILS_DASK_LARGE_DATA_SHAPE=8192 \
GEOUTILS_DASK_LARGE_DATA_MEMORY_LIMIT=512MB \
pytest -large-data -m large_data -ra