How imPress is tested: the single test runner, every suite, the CI pipeline, the test strategy, and the hardware bench checklist. References: docs/guides/testing.md, docs/testing/TEST_STRATEGY.md, docs/testing/HARDWARE_VALIDATION.md.

• Running the tests
• The suites
• What each layer catches
• End-to-end and fault-injection tests
• Continuous integration
• Rules for new tests
• Hardware validation

python3 -m venv .venv && . .venv/bin/activate
pip install -r backend/requirements-dev.txt
python run_tests.py                        # every default suite, about a minute
python run_tests.py --list                 # what exists
python run_tests.py -s backend -s host     # selected suites (prefix match)
python run_tests.py --with-ui              # + Playwright browser tests
python run_tests.py --with-idf             # + ESP-IDF builds of all three projects (Docker if no IDF)
python run_tests.py --with-frontend        # + the React build
python run_tests.py -s backend --coverage reports   # + coverage
python run_tests.py --json report.json --markdown report.md

The runner prints a report with each suite's status, counts and duration, the slowest tests, and details of every failure.

Runner behaviour:

• Progress: long suites show a live status line (elapsed time and the latest output), plus a heartbeat in CI.
• Missing packages: if the interpreter lacks a package a suite needs, the suite fails once, with the exact pip install command, instead of hundreds of import errors.
• Interrupting: Ctrl-C stops the whole process tree cleanly, prints the report, and exits 130. --timeout SECONDS bounds every suite.
• Unavailable tools: a suite whose tools are missing (no C compiler, no Docker) is reported as SKIP with the reason.

The latest full local run, on varun/v2.1 at a1662c2 (2026-10-09, run_tests.py --with-ui): 666 passed, 0 failed, 0 skipped in 2 min 15 s. Backend line coverage: 88 % (measured before the #66 work).

backend/tests/gateway_sim.py is a simulated gateway. It follows the firmware's documented behaviour:

• it registers and sends heartbeats;
• it buffers mesh messages and posts them in batches, keeping a batch on a 5xx response and dropping it on a 4xx, as the firmware does;
• it runs the OTA client's steps, with optional faults: a corrupted download, power loss after a step, or an image that fails its health check.

Fault injection found a real bug: a database error on commit used to be an unhandled 500. It is now a 503 with nothing saved.

.github/workflows/ci.yml runs on every pull request, on pushes to the long-lived branches (main, v2, v3, varun/**), and on demand.

Pipeline properties:

• Least privilege: the workflow can only read the repository.
• Pinned supply chain: every action is pinned to a commit SHA, the ESP-IDF image to a digest, and Python packages to a hash-locked set (backend/requirements-lock.txt). Frontend packages come from package-lock.json (npm ci).
• Cancellation: a newer push to the same branch cancels the running pipeline.
• Self-checking: tests/test_ci_workflow.py checks the workflow itself. Every job must have a timeout, every action must be pinned, every runner suite must be wired into CI, and the result gate must depend on every job.

The README shows one CI badge, for varun/v2.1. GitHub's badges are per workflow, not per job.

.github/workflows/release.yml publishes releases and the backend image (docs/guides/releases.md):

It never moves a tag or replaces a published release. tests/test_release.py checks its status rules (against a fake GitHub), its safety settings, the Dockerfile and the release notes.

1. Test behaviour through the API. Touch the database only to create states the API can't produce (time passing, a crash) or to check stored rows.
2. Never block forever. WebSocket tests broadcast a marker after the action and read until it arrives. Never sleep waiting for something.
3. Prove the test can fail: break the code it guards once (a mutation) and watch it fail before trusting it.
4. One source of truth across the boundary. When firmware and backend must agree, the test reads both (the frame fixture, schemas.py, partitions.csv, health.py) instead of copying a value.
5. Fix flakiness, don't retry it. A test that fails intermittently is a bug in the test or the code.
6. Simulation is not hardware. A passing simulated test never counts as hardware verification.

Writing a backend test:

from conftest import DEVICE, auth, login, make_class

def test_my_rule(client, db):
    h = auth(login(client))
    cid = make_class(client, h, activate=True)
    r = client.post("/api/device/register", headers=DEVICE, json={"mac_address": "48:F6:EE:00:00:01", "device_type": "c6"})
    assert r.status_code == 200

client starts the app on a fresh temporary database; db(fn) runs an async ORM function against it. Helpers: login, auth, make_user, make_class, make_student, upload_firmware, DEVICE (device-key headers), and gateway_sim.Gateway.

As of v2.1, none of these checks has been run on real boards. Each check has a test issue with numbered cases, expected results, edge cases and pass criteria; the tracker is #94. The issues are in milestone v2.1.0 with the testing label, so the 2.1.0 release stays Beta until they are closed. Each check below has a procedure and pass criteria in docs/testing/HARDWARE_VALIDATION.md. Record a result there as passed | failed — <boards>, firmware <version + commit>, backend <version>, <date>, <who>; <notes>.

The v2.1 security work adds checks to run on hardware too:

• Device keys: enrolment, and reset with re-enrolment.
• Signed firmware: a signed device accepts a signed image and refuses an unsigned one.
• Device TLS: a TLS handshake from a gateway and from a hub.