Experiment formal installation a12 stimulus run 01 - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/formal/installation-a12-stimulus-run-01.md. Original SHA-256: 5b9f63d22eb59091e5df17babb74ea9a39f2bf079bf58c7e50f2690086c4fc50. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

A12 stimulus diagnosis, run 01

Status: diagnostic counterexample and host-side fixture probe completed on 2026-10-09. This is not an installation acceptance run. At the time of this diagnostic, the revised revision-2 parallel fixture had not run in a Debian VM.

Subsequent evidence: the fresh unified VM baseline ran the parallel fixture and passed A12 with actual kernel OOM events and systemd oom-kill. This diagnostic's historical failure and host probes remain unchanged; the fresh run subsequently rejected all three required mutation controls.

Question and fixed contract

The first unified-cgroup installation VM run failed A12 because both memory hogs reached RuntimeMaxSec=300 before MemoryMax triggered an OOM kill. The question was whether the registered stimulus was blocked by MemoryHigh throttling and whether concurrent private-page faults could reach the same hard caps without changing the cap, high threshold, task limit, deadline, service-health checks, or OOM-result rule.

The VM's registered limits remain node MemoryHigh=128M, MemoryMax=256M, TasksMax=128, and slice MemoryHigh=2816M, MemoryMax=3328M. The required unified-cgroup result remains Result=oom-kill before the existing 300-second deadline. The positive control and local probes below do not change those acceptance criteria.

VM counterexample

Run readiness-debian13-01 used source snapshot bc858a7580a4e89994d744d4b3780b2979a3520f, package SHA-256 ea4239365e0d96d38a4aae91d16307693481beb4485b9b7828ba59faaa439ea5, and the historical default tail /dev/zero stressor. It ran under QEMU TCG with a Debian 13.7 guest, systemd 257 as PID 1, unified cgroup v2, 6 GiB guest memory, and an 8 GiB virtual disk. The acceptance script SHA-256 was fed39f6502864c872ec1dedda001d859cb81db21270403d743aad3e5d8c23053.

A1–A11 and A13a/b passed; A8h observed the configured 128 MiB node and 2816 MiB slice memory.high values. A12 alone failed:

Hog Result Peak Hard limit
Node timeout / 15 144,203,776 B 268,435,456 B
Slice timeout / 0 3,074,211,840 B 3,489,660,928 B

The task control reached 128 tasks and logged five fork failures. Both services stayed active with unchanged restart counts, and delivery continued after pressure. The failure is therefore the memory stimulus: neither tail hog caused an OOM event before its deadline. It does not show a service-health or task-limit defect.

Receipt and full acceptance/journal logs: $FABRIC_DATA/results/installation-qemu/readiness-debian13-01/. Its cleanup-followup.json records removal of the owned guest scratch on 2026-10-10 after provenance, process and mount checks; the failed acceptance evidence remains.

Bounded host probes

Both probes ran through python3 tools/resource_group.py --delegate --runtime-seconds 120, with the complete parent tree under the 20 GiB outer memory cap and zero swap. They created and removed a dedicated child cgroup; neither ran project tests or changed source files.

The first probe tested a 32 MiB MAP_POPULATE control and a 192 MiB single mapping under child memory.high=50 MiB, memory.max=128 MiB, zero swap, eight tasks, and half a CPU. The control stayed resident at 33,816,576 B with no pressure events. The large mapping stalled for 20 seconds at a 59,322,368 B peak; high increased by 391 while max, oom, and oom_kill remained zero. The harness killed it after timeout. This candidate did not reach the hard cap. Receipt: $FABRIC_DATA/results/installation-qemu/a12-memory-probe-01/receipt.json.

The second probe retained the same positive-control result and used 96 gated workers, each dirtying and holding a private 2 MiB anonymous mapping, for 192 MiB aggregate requested memory. Child limits were unchanged at high 50 MiB and max 128 MiB, with zero swap, 128 tasks, and one CPU. The leaf reached exactly 134,217,728 B. Before the 45-second probe deadline, memory.events had increased by 8,830 high, 241 max, 10 oom, and 2 oom_kill; the main supervisor was itself an OOM victim before all workers reported ready. A later final snapshot showed 8 cumulative oom_kill events, so six additional kernel OOM kills occurred after the pressure snapshot and before final sampling. Harness cgroup.kill was used only for cleanup and does not increment oom_kill. All probe cgroups were removed. Receipt: $FABRIC_DATA/results/installation-qemu/a12-memory-probe-02/receipt.json.

These host measurements support parallel private-page faults as a stimulus candidate, but do not establish Debian systemd Result=oom-kill behavior or VM acceptance. The exact test helper and its source hash are recorded at runtime by the revised acceptance harness.

Revision-2 fixture and decision

Revision 2 preserves tail as the acceptance script's default. The unified Debian VM rerun explicitly chooses parallel. It first requires a resident 32 MiB positive control below MemoryHigh, then runs 96 workers per hog. For hard limit L and page size P, a worker dirties and holds P * ceil(ceil(2L/96)/P) bytes. The per-worker allocations are rounded up to pages and the aggregate request exceeds twice the same hard limit. The transient unit keeps the original task cap and 300-second deadline; OOMPolicy=kill ensures the unit is terminated as a group after a kernel OOM victim. The acceptance still requires bounded peak usage, systemd's OOM result, task-cap evidence, service health, and continued delivery.

The runner verifies the transferred helper hash and records the explicit parallel selection and helper hash in its receipt. A VM run of revision 2 is the next evidence required; neither host probe is counted as a pass.