Status: IN PROGRESS. Kept as a running log because this investigation produced more retracted numbers than final ones, and the retractions are the useful part — each one came from a measurement-harness defect that looked exactly like a real result.

The two clips recover 45% of the int8->W4A4 NLL gap and cost nothing in storage or compute. Independently reproduced later by bisect on a 255-token screen: 37.04 (pre-clip) -> 24.53 (post-clip), -33.8%. Direction and rough magnitude agree across two different samples and two binaries.

Rejected with numbers:

• fp32 GPTQ factorisations: +7.2% PPL for 1.5x pack speed. Gated OFF, measurement recorded in the gate.
• per-group int4 scales: -1.26% PPL for +5.7% pack size. Poor trade for a memory-motivated format.

ORK_W4A4_DIAG splits a weight's matmul error into its weight and activation halves. Measured on 14 weights: mean A/W = 0.97 (range 0.75-1.19) — parity, not the activation dominance predicted. The two are INDEPENDENT: they add in quadrature (predicted 0.2074 from the halves, measured 0.2066).

Consequence: neither half alone gets near int8, and improving one by X% moves the total by ~half that. This is what killed per-group scales before they were built (weight-side only, capped at ~35%) and what promoted activation clipping to first place.

Later refinement: the ranking that motivated "selective per-layer precision" came from this diagnostic measured against RTN weights. The pack's own qerr metric (diag(H)-weighted, against FINAL GPTQ weights) disagrees and shows the distribution is nearly FLAT — ranks 3-16 span 0.1980..0.1882, under 5%. Only two attn_output weights stand out. Selective per-layer precision has a weak premise on this model.

Measured on the 255-token screen (all offline, same binary, paired):

All-rotated is fine. All-unrotated is fine. Mixing is worse than either. The unrotated variant added MORE precision and was 7x worse, so rotation — not precision — dominates. Rotation accounts for ~85% of the mixing penalty.

Corroborated by the QuaRot literature, which inserts rotations in matched pairs across the residual stream — though NOTE the mechanism does not transfer directly: ork-driver's rotation was verified local (every activation site rotates a private copy, memcpy then FWHT; src1 is never mutated), so there is no unmatched rotation to explain it. The mixing regression is still unexplained.

Also measured: rotated i4a8 (rotated int4 weights, int8 activations) has error equal to the WEIGHT-ONLY error — 0.1578 vs W-only 0.1576, three decimals, every weight. int8 activations remove the activation half entirely. Mean -28.6% vs W4A4 at zero storage cost.

i4a8 caveat (why it is not a free win): it inflates int4->int8 into IOVA, and under multi-domain streaming that repeats per domain swap. Measured inflate throughput: 1.44 GB/s scalar, 21.8 GB/s NEON (15.2x, bit-exact) — a 0.4 GiB int4 model inflates in 59 ms with NEON vs 898 ms scalar. Against a 512-token prefill (int8 2.84 s vs W4A4 4.79 s), int8+inflate = 2.90 s, still 1.65x faster than W4A4 even re-inflating the whole model every forward. Affordable IF the inflate loop is NEON.

• orkpack v6: the PACK decides each weight's route (DT_I4_NATIVE / DT_I4 / DT_I8), not a global env knob. Entry gains qerr — the diag(H)-weighted relative error measured at GPTQ finalize — so re-tiering is a pure function of the file (ORK_PACK_RANK prints it worst-first with the MiB each promotion costs).
• v7 (IN PROGRESS): DT_I8_ROT, a rotated int8 tier, so promotion no longer leaves the rotated regime. NOT WORKING YET — GPTQ finalize re-packs unconditionally at int4 and ignores the width, so the pack comes out byte-identical to pure int4 and any PPL from it is meaningless.
• Off-board pack builds: 12.9 min -> 1.2 min (10.8x) on an M5 Max VM, validated equivalent (VM-built pack 17.021 vs board-built 17.332, full run). Enabled by an offline no-device context in ork-driver.
• ppl_screen.sh: two-tier PPL. ~40 s screen for iteration (tracked the full run to 0.16%), --full as the release gate. Now prints binary build time + backend hash on every run.
• ORK_MM_CHECK (one matmul vs an exact reference) and ORK_ACT_TRACE (residual-stream fingerprint per matmul, to localise divergence).

Recording these because every one looked like a real result at the time.

The pattern: every wrong conclusion came from the measurement apparatus, not the model. Three fixes now in place — --no-t on sync, provenance printed per run, and a disk check before trusting a pack.

The rotated int8 tier is implemented end to end — dtype, v7 bump, width plumbing through weight quant, activation quant, MAC selection, persist and load, and finalize taught to skip 8-bit weights (it was re-packing them at int4 unconditionally, which is why the first "rotated w8a8" number was meaningless).

Correctness of the WRITE is confirmed by arithmetic: a 2-layer promoted pack is 163,446,928 bytes vs the pure 161,349,776 = +2,097,152 = exactly 2 x (2 MiB int8 - 1 MiB int4), with 2 DT_I8_ROT entries and finalize reporting 100 quantized rather than 102.

But it has no trustworthy PPL, for two separate reasons found while trying to measure it:

1. ork_i8_mm_load has NO offline branch (only ork_i4_mm_load got one), so offline a DT_I8_ROT entry cannot be read; the run silently falls back to cold-quantising from source, and at scoring time the build-time promotion check does not apply — so it comes back as rotated INT4. The offline 18.0397 is therefore not measuring this tier.
2. The board and the offline path disagree by 54% on the SAME pack (v7 pure: 15.9011 offline vs 24.5288 board, same tokens, pack confirmed read with "loaded 102 from disk"). Until that is resolved no cross-machine number means anything, and the offline path cannot be used to evaluate quality.

Note this divergence was seen much earlier, attributed to a stale binary, and dismissed. It is real.

Also unresolved and now suspicious: v7 pure scores 24.5288 on the board where the v6-era pure pack scored 17.2268. Same tier, same text. Either the pack build changed under the recent edits or the board path did.

Byte-level dump of one matmul (ORK_MM_DUMP), same pack, same tokens, board vs offline:

Activations, their scale, rotation and clipping all agree exactly. The WEIGHTS differ — and the scales differ by a CONSTANT 1.28000 across every channel. 1.28 = 1/0.78125, and 0.78125 is exactly the last value of the GPTQ weight-clip grid (1.0 - 0.03125*7).

So the board's weights lack the GPTQ clip: it is silently re-quantising from SOURCE via the cold RTN path while reporting loaded 102 from disk. Higher error (0.168 vs 0.111) and far worse PPL (24.53 vs 15.90) follow directly.

Consequences. Board-measured W4A4 numbers in this configuration were RTN, not GPTQ. The offline path was correct all along — the "offline diverges from the NPU" conclusion was backwards, and the VM screen is the one that was telling the truth. Any board W4A4 figure taken while this was live needs re-checking against whether its pack actually reached the matmul.

Method note. Five hypotheses (MAC precision, tiler, un-tiler, zero-copy import, activation cache) were each eliminated by inference over several hours. One byte-level dump of the intermediates localised it in a single run. Hash the stages first next time; a constant ratio in a scale array names the culprit immediately, where a perplexity delta never can.

5c was wrong. The board was NOT running RTN weights. The pack file on the board differed from the one on the VM — same size, different sha256 — because a transfer reported success and was never verified. The constant 1.28 scale ratio that looked like a smoking gun (1/0.78125 = the GPTQ clip grid) came from a corrupted file, not from a missing clip.

After re-transferring WITH hash verification, every stage matches bit-for-bit:

The NPU int4 MAC and the offline CPU GEMM are BIT-EXACT with each other. The offline path is a faithful emulation — the original claim was right; the retraction was caused by the corrupted file. And the board's recurring 17.2268 was always the correct number; 24.5288 was the bad pack.

ORK_ACT_TRACE on both, same pack: all 102 matmuls agree, first divergence 2e-7 (fp noise between different CPU builds). model / pack / text sha256 all verified identical.

Residual: 17.2268 (board) vs 15.9011 (offline), NLL 2.846 vs 2.766. With every matmul bit-identical, this must come from ops the trace does not cover — the board's backend can claim NON-matmul ops (norms, SDP, softmax) and run them on the NPU at reduced precision, where the offline context has no device and they fall back to fp32 CPU. Testable by comparing which ops each backend claims. NOT yet confirmed.

Process failure worth naming. Four separate conclusions in this session came from unverified inputs: a stale binary (mtime), a truncated pack (full disk), and a corrupted pack (unverified transfer). Every one looked like a real result and consumed hours. Inputs are now hashed before use; that check costs one command and would have prevented all of them.

Final answer to the board-vs-offline 8.3% (17.2268 vs 15.9011), reached by dumping every stage of one matmul on both machines, per ROW.

The decisive observation was that row 0 is not enough. Both the dense and the chain matmul are bit-identical at row 0 — which is what made them look exonerated. Dumping later rows:

And at chain row 1 the difference is in the INPUTS, not the arithmetic:

ai        board 56ce0b32..   VM 921e87c4..
as[row]   board 0.0231712889 VM 0.0231734682     <- differ by ~1e-5

The mechanism, end to end:

1. ggml's CPU kernels differ slightly between an A76 build and an M5 build (feature-gated vec_dot selection). Measured with ORK_OFF=1, pure fp32, no ork at all: board 11.7845 vs VM 11.7783 — 0.05%.
2. blk.0-2 are SSM layers; blk.3 is the first full-attention block, which is exactly where the first divergence appears (op #21, blk.3.attn_output). That is where the CPU-side attention difference enters the ork path.
3. 4-bit quantisation amplifies it. A 1e-5 shift in a row's mx flips values sitting near a quantisation boundary, and with only 16 levels a flipped decision is a large relative change in that element. The activation hash changes wholesale.
4. It compounds with depth: 0.0086% at op #21 -> 7.0% at op #77 -> 8.3% in final PPL.

Pure CPU shows only 0.05% because there is no quantiser to amplify anything.

NOTHING IS BROKEN. Not the NPU MAC, the chain path, the loaders, the offline emulation, or the pack. Two correct implementations disagree because the format is chaotically sensitive to its inputs.

• Cross-machine ABSOLUTE PPL is meaningless for a 4-bit model — not merely unreliable. Two correct implementations will differ by several percent.
• Same-machine A/B is exactly valid, and is what every quality conclusion in this log rests on.
• int8 is far less sensitive (127 levels, boundary flips rare) — hence the 0.05% pure-CPU agreement.
• Corollary: a W4A4 perplexity number is only meaningful with its machine, binary and pack stated. Two W4A4 numbers from different hosts cannot be compared even when both are correct.

5c ("board runs RTN weights") — wrong, it was a corrupted pack transfer. 5e ("isolated to the NPU chain path") — the chain path is where it SURFACES, but the chain arithmetic is correct; the inputs differ. Kept below for the elimination table, which is still accurate.

Board 17.2268 vs offline 15.9011 on a hash-verified identical pack, model and text. Elimination:

What remains is the 48 CHAINED matmuls, the only ork-executed path that was never instrumented (the 150-op trace that finally covered them shows the first real divergence at op #21, blk.3.attn_output — the output projection of the first full-attention block — growing to 7.0% by op #77).

On the board those run the NPU hardware chain; offline they run orki_cpu_chain_i4's exact int32 GEMM. So the offline path's fidelity is PATH-DEPENDENT: bit-exact for the dense MAC, optimistic for chained matmuls. That is the correct summary of what the offline screen can and cannot be trusted for.

Consequence for methodology: offline A/B comparisons remain valid (both arms run the same paths on the same machine); offline-vs-board ABSOLUTE comparison is not, until the chain path is either emulated faithfully or excluded.

Useful by-product: pure-CPU fp32 reference is 11.78 on this text/window — the true ceiling. int8 11.95, W4A4 15.90 (offline) / 17.23 (board).

The rotated int8 tier is complete: dtype (v7), width through weight quant / activation quant / MAC selection / persist / load, and GPTQ finalize taught to skip 8-bit weights (it had been re-packing them at int4, which made the first attempt meaningless). Write correctness confirmed by arithmetic: a 2-layer promoted pack is +2,097,152 bytes = exactly 2 x (2 MiB int8 - 1 MiB int4), and the run reports W4A4=98 i4a8=0 W8A8=2.

Board, same machine, hash-verified packs, 255-token screen:

Higher precision, worse perplexity — and per 5f that is expected noise, not a result. Promoting two layers perturbs downstream activations, which flips quantisation decisions in the other 98 W4A4 layers. The three mixing experiments run so far land at +0.41% (rotated per-group), +2.8% (unrotated int8) and +4.2% (rotated int8): all "worse", scattered, with no ordering by precision. That is a noise band.

Same-config runs are exactly reproducible (v6pure and v7pure both 15.9011), so this is not run-to-run variance — it is a genuine chaotic response to ANY perturbation, and 255 tokens cannot resolve an effect smaller than it.

Methodological consequence: selective per-layer precision cannot be evaluated by single-config perplexity at screen sample size. Either use the full text (24x the tokens) or, better, evaluate many promotion sets and compare distributions — a single A/B of one promotion set is measuring chaos.

This also retro-explains every confusing mixing result in sections 5b-5e: they were all sampling the same noise band, which is why more precision kept coming out "worse" in no consistent order.

1. Why does mixing degrade below both pure tiers? Not disk, not truncation, not the offline path, not the activation cache (type-guarded), not weight corruption (non-promoted weights bit-identical across packs; promoted ones measurably better), not error bias (+0.095 vs -0.064), not rotation escaping the matmul (verified local). ORK_ACT_TRACE localises the first divergence to the op immediately after the first promoted weight, growing to 5.4% two layers later — but divergence from the pure run is NOT error, since the promoted layer is genuinely more accurate. Next instrument: an fp32 reference forward via ggml_backend_sched_eval_callback (fires for every node regardless of backend), so activations can be compared against truth rather than against each other.
2. DT_I8_ROT is built but unmeasurable — needs an offline branch in ork_i8_mm_load, and needs the board/offline divergence resolved first (see 5b). 2b. BOARD vs OFFLINE differ by 54% on one pack. This is now the top blocker: it invalidates the fast offline screen for quality work, which was the whole point of building it.
3. int8 via pack (13.23) vs int8 from source (12.03) — a ~10% same-tier round-trip gap, unexplained.
4. ork_i4_mm_run_grouped is ~13x off its own achievable dispatch cost, and ~2.9 ms per int4 submit at M=128 is ~17x the documented 167 us floor — a tax on EVERY int4 matmul, unrelated to any of the above.

RETRACTED 2026-08-23. The tier table in this section measured three bugs, not the tiers. The pack-clobber findings and the hash-provenance rule below remain valid; every PPL number in the table does not. Corrected numbers in 5L.

ORKPACK_VERSION was bumped to 8 for DT_I4_ROT_A8, and the "additive versions stay readable" relaxation was applied to the staleness gate only. Two other sites — the adopt gate and ggml_backend_ork_orkpack_valid — kept a strict f.version == ORKPACK_VERSION. The three gates then disagreed in the worst possible way:

• staleness gate: pack is fine (so it is not marked for rebuild)
• adopt gate: version != 8, so the index is loaded but the pack is not adopted
• control falls through to the WRITE path, whose contract is "missing/stale pack self-heals"

The pack was therefore overwritten with freshly-computed RTN content, and the run reported the RTN number as if it were the pack's. Two GPTQ packs (v7pure, ra8k28) were destroyed this way; both came back byte-identical to each other (same env, no promote list ⇒ same RTN output), which is what exposed it: two packs with different contents cannot legitimately produce a PPL match to six figures.

Diagnostic that settled it in one command: ls -la + sha256sum on the board packs. The two failing arms were root-owned and stamped with the run's timestamp; the two arms that reproduced exactly were older and untouched. Provenance beat every code-reading hypothesis.

Two fixes, both landed:

1. ork_pack_version_ok(v) = (v >= 6 && v <= ORKPACK_VERSION) — one predicate, all three call sites.
2. A no-clobber guard: reaching the writer with a pack that is well-formed and NOT stale is by definition a loader-gate disagreement, so it now abort()s with a diagnostic instead of rebuilding over the file. ORK_ORKPACK_CLOBBER=1 is the deliberate override.

Guard verified: post-run sha256sum of both packs is unchanged.

Rule this establishes: never infer pack identity from the PPL. Hash the pack before and after every run. A silently-rebuilt pack is indistinguishable from a quality regression, and this is the second distinct mechanism this session that produced exactly that illusion (the first was rsync -a preserving mtimes so cmake skipped the rebuild).

Control re-run after restoring both packs from the VM, hash-verified: 17.2268, exact. Board, same machine, 255-token screen, k=28 = attn_output + ffn_down promoted:

Rotated i4a8 is the memory-constrained answer. It matches rotated int8 to +0.27% PPL while costing 24% less disk, because the promoted weights stay int4 ON DISK and are inflated to int8 containers at load — only the activations and the MAC widen. Against plain i4a8 at identical 161 MB, rotation buys -7.4%, which is the same "rotation is not optional" result as section 3, now confirmed at the i4a8 width rather than only at W4A4.

Note this is a 10.5% separation, well outside the ~±4% chaotic noise band established in 5g — unlike the 2-layer promotions, this one is resolvable at screen sample size. The ordering is also now monotonic in effective precision for the first time (W4A4 < i4a8 < rot-i4a8 ~ rot-int8 < int8), which is what 5g predicted would emerge once the effect exceeded the noise.

Route tally caveat: DT_I4_ROT_A8 weights are counted in the W8A8 bucket by the pack-driven route counter (they take the int8 MAC), so ra8k28 reports W4A4=72 i4a8=0 W8A8=28. The bucket is coarse; it is not evidence the weights were stored at int8.

The Hessian is K*K doubles per weight and it is not disk-backed — unlike a weight, an evicted H cannot be paged back, only rebuilt by replaying the whole calibration corpus. Registering every weight in one pass costs sum(K^2*8) resident, which for Qwen3.6-27B (64 layers, n_embd 5120, n_ff 17408) is ~236 GiB — the 64 ffn_down at K=17408 are 2.42 GiB each. The VM has 62 GiB. Single-pass GPTQ is not merely slow at this scale, it is impossible.

So: calibrate a LAYER RANGE per pass — set window, replay corpus, finalize, repeat. Peak residency becomes the window's Hessians rather than the model's. ORK_GPTQ_WINDOW=<layers>, or derived from ORK_GPTQ_MEM_GB (default 24) against the largest registered K.

This is NOT the same as domain/page-cache windowing, and the distinction is the whole design:

Every wrong version still exited 0 and still wrote a valid 102-weight pack. The only thing that exposed them was a same-machine A/B against a known baseline.

1. H allocated in the wcache-MISS branch. That branch runs once per weight — the forward that first packs it. By window 2 every weight is cached and never returns, so only window 1 could ever claim. Log: window [0,8) -> 102 quantized, windows 2-3 found nothing.
2. Claim moved to the every-forward path, but "no window set" still meant "claim all". The discovery decode runs before any set_window, so it allocated all 102 Hessians before window 1 began. Same result. Fix: THREE states — UNSET (discovery: register metadata, claim nothing), RANGE, and ALL (= [0,n_layer), no special case). The driver must therefore always set a window and always run its claim decode.
3. gptq_rows() read an arbitrary map entry. Fine when every entry was claimed; with windowing the map also holds unclaimed weights (samples 0 → the row loop runs the corpus dry) and retired ones (samples from THEIR window → the loop exits instantly and later windows calibrate on the claim decode alone). Plus samples was never reset per window.

Working version: window [0,8)/[8,16)/[16,24) -> 34 quantized each, 7 calibration batches per window.

Same machine (VM, offline), 256-token screen, 0.8B:

One window through the new code reproduces the old single-pass number EXACTLY, so the windowing machinery is clean and the +9.7% comes from splitting, not from the rewrite.

ORK_GPTQ_SEQ=0 is a no-op — do not trust that row. Releasing a wcache_pin only makes an entry eligible for eviction; nothing evicted, so the GPTQ'd weights stayed cached and both windowed arms were sequential. The knob needs to actively restore RTN codes between windows to mean anything. Until then the sequential-vs-one-shot question is open, and the 9.7% is unexplained.

Leading hypothesis (untested): our calibration forward is W4A4, so activations are 4-bit too. Each window's H is then built on activations already degraded by upstream weight AND activation error, and it compounds. Textbook sequential GPTQ assumes clean activations through quantised weights. Do not repeat the claim that sequential is "strictly better" — that is true of the textbook formulation and is currently contradicted here.

Single-pass is unavailable at 27B, so the real comparison is windowed vs RTN: 17.4465 vs 26.2782, a 34% improvement. Windowed GPTQ ships for 27B. The 9.7% given up against an unattainable baseline is a follow-up, not a blocker.

A packed run opens BOTH the source GGUF and the .orkpack, so every matmul weight is resident twice. At 27B that is 28.6 GB + 12 GB = 40.8 GiB against 31 GiB of board RAM.

It does not fail as an OOM. The GGUF is mmap'd, so pages churn: submits time out, the driver self-heals, and dmesg fills with RKNPU: soft reset every ~100 ms. It reads as a driver fault. The same 27B command succeeded once (PPL 10.6780) and failed once, and the difference was page-cache state. Several hours went into chasing it as a shape/M-ceiling bug before the working-set arithmetic made it obvious.

ork_write_stub_gguf emits <pack>.gguf at pack-build time (or ORK_MAKE_STUB=1 from an existing pack — the stub needs only the source model and the pack's tensor NAMES, so rebuilding 12 GiB for its sidecar is unnecessary). Header + KV + tensor-info are copied verbatim, so every tensor keeps its exact data offset and llama.cpp needs no change; only tensors the pack does not own are copied into the data section, and the rest are left as file HOLES.

PPL through the stub is identical to the full GGUF (26.2782 both, 0.8B). 27B working set: 40.8 → ~14.6 GiB.

Guard: the holes read as zeros, which is correct only while the pack covers them, so ork_stub_verify aborts on a tensor that is all-zero AND absent from the pack — the signature of a stub run without (or with a mismatched) pack. A stub is not a model on its own. ORK_NO_STUB=1 skips emission.

Retiring the GGUF entirely is roadmap Tier 15 (orkpack-native loader).

The result. Pure W4A4 (RTN pack, no GPTQ), stub gguf, clean board, W=64: PPL = 10.6780, 400/400 weights loaded, 0 soft resets — reproduced exactly across a reboot. The whole path holds at 34x the parameter count anything had been run at: offline pack build (400 weights, 11.6 GiB, 2 minutes), multi-domain residence, board execution.

Model shape matters for reading everything below — Qwen3.6-27B is a hybrid: 64 layers, but only 16 are full attention; the other 48 are Gated Delta Net (ssm_*, and the loader reports "fused Gated Delta Net enabled"). So attn_output exists 16 times, ffn_down 64 times. n_embd 5120, n_ff 17408.

The attn_output,ffn_down promotion to rotated-i4a8 — the tier that WON at 0.8B (15.4138, section 5h) — loads zero of its 80 promoted weights at 27B:

400 - 320 = 80 = 16 attn_output + 64 ffn_down — exactly the promoted set, no more and no less. Each one misses ork_persist_load_i4native, silently falls back to inline packing, and the model outputs garbage.

The failure is silent by construction, which is the part worth fixing beyond the bug itself: a pack-miss is a legitimate event (that is how a cold weight gets built), so serving 80 weights from a different tier than the pack specifies produces a confident wrong answer rather than an error. A read-mode miss on a weight the pack demonstrably contains should be loud.

The 0.8B rot-i4a8 result stands on its own; it simply does not carry to these shapes yet.

BCHAIN requires Sn==1. At 27B:

41,648 int4 submits for 64 tokens. At W=256 the per-row chain overruns preallocated scratch — src/npu/core/sched.c:250 gives each core 64 KiB regcmd, 64 KiB task descriptors and a 256 KiB A-stage, sized for the M=1 decode path. At K=5120 the A-stage holds ~51 rows/core, so ~86 rows/core refuses: ork_i4_dyn_begin_mc returns NULL -> -4 -> ORK_RC_WEDGE_PRONE -> graph_compute … -1.

This is NOT a hardware limit and should not be recorded as one. The silicon envelopes are per-TILE (mg_max*64 int8, orki_f16_mcap fp16, bank-width 131072/K int4) and tiling loops over them. Fix is to grow the scratch or chunk M instead of refusing. Until then a representative 27B prefill benchmark is not obtainable — the per-row path is not the intended one and its throughput should not be quoted.

Signature, and it is unmistakable once seen:

PRIME_FD_TO_HANDLE: Cannot allocate memory                     <- dma-buf import ENOMEM
CREATE FAIL: errno=14(Bad address) dom=0 dom_iova=1085MB ceil=3900MB   <- "full" at 28% of ceiling
CREATE FAIL: errno=14(Bad address) dom=1..11 dom_iova=36MB             <- every later domain, instantly

PRIME_FD OOM plus domain-full while system RAM is free. Preceded in dmesg by WARNING … __iommu_dma_unmap in rknpu_gem_object_destroy -> iommu_dma_unmap_sg.

Recovery is a reboot (SSH stays alive; sudo reboot, then re-pin governors — DDR comes back as dmc_ondemand).

Cause: I terminated two in-flight 27B runs with SIGTERM. AGENTS.md warns specifically against kill -9, and I read that as SIGTERM being safe. With ~12 GiB imported across 12 domains it is not: GEM teardown fails to unmap and every subsequent run inherits the leak.

This cost hours and, worse, corrupted the diagnosis — the identical W=64 command returned 10.6780 before the kills and failed every time after, so the same shape looked non-deterministic and the failures got misattributed first to an M ceiling and then to memory pressure. Both were wrong; the board never recovered.

Rule: let a multi-domain run finish or hit its timeout. Do not terminate it. If one must be stopped, budget a reboot before trusting any measurement that follows.

The stub gguf (section 5j) took the working set from 40.8 GiB to ~14.6 GiB and is correct and worth keeping — but it did not fix these runs, because the leak was the actual cause. Recording that explicitly so the stub is not remembered as the fix for bug 3.

1. The loader could never match the writer. ork_w_dump writes an int4 weight in the i4-NATIVE TILED layout — sum(pgup(Kp*Nc/2)), page-padded tiles, no header, no scales. The DT_I4_ROT_A8 load path called ork_i4a8_mm_load, which wants the COMPACT container — hdr + bscale[N] + Bi4[K*N/2] — and gates on if (n != hdr+sc+nib) return NULL. Those sizes cannot coincide, so every DT_I4_ROT_A8 weight ever written returned NULL. Not a 27B problem: it had never worked, at any size.

A pack MISS is a legitimate event (it is how a cold weight is built), so the fallback to inline packing is silent by construction. At 0.8B it produced a plausible wrong number. At 27B it inline-packed 80 weights at ~27-89 MB each and exhausted IOVA.

2. The routing never read the pack's rotated bytes. want_i4n = (dt == ORKPACK_DT_I4_NATIVE), so DT_I8_ROT and DT_I4_ROT_A8 both went to mul_mat_i8 — which re-quantises from the SOURCE tensor. A rotated weight is only correct against rotated ACTIVATIONS, and only mul_mat_i4_hadamard applies them. So the promoted weights were running as unrotated int8 re-quantised from source, and the pack's rotated bytes were dead weight on disk. With a stub gguf that source is a file hole, which is how the 27B arm scored 136986.

3. Activation width was not in the reuse key. The hadamard path quantises activations to the tier's width (4 for W4A4, 8 for the rotated tiers) and a MIXED pack has both sharing one activation row at the same (y, M, K); whichever weight ran first won. last_abits added. UNVALIDATED — do not credit it with anything: it changed nothing measurable on either rotated pack (identical to 4 decimals before and after). Kept on correctness grounds only.

Fixes: ork_i4a8_mm_load_tiled (ork-driver 5a61bf0) + routing/key (fork a645cdc21). Fixed in the loader, not the writer, so existing packs work with no rebuild. The un-tile walk is FACTORED into orki_i4_untile_blob and shared with ork_i4_mm_load — a tiler and un-tiler drifting apart is silent, and this bug is what that costs.

Rotation at i4a8 width HURTS (15.7034 vs 15.3087) — the exact reverse of 5h's headline. That claim was an artifact of the promoted weights running unrotated-from-source.

DT_I8_ROT is a separate, still-open bug. 21.1476 is worse than pure W4A4, which is impossible for a strictly-higher-precision tier. Its loader (ork_i8_mm_load) does match its blob format, so this is NOT the same format mismatch. Its old 15.3717 was equally invalid, so the tier currently has no valid number.

Void — every mixed-tier number in this document before this section, including the 5h table, the rot-int8 vs unrotated-int8 comparison, and the k=0/2/5/23/28/100 promotion sweep. They measured combinations of bugs 1-3.

Survives — the pure arms (they never mix activation widths): pure W4A4, stock int8, CPU fp32, the activation/weight clip results (sections 1-3), the 27B pure-W4A4 result (10.6780), the windowed-GPTQ comparison (5i, all-W4A4 packs), and the stub gguf equivalence (5j).

The only mixed-tier win that survives is unrotated i4a8 at -3.7%.

A silent fallback that produces a plausible number is worse than a crash. Four distinct mechanisms manufactured believable wrong results in one session: a skipped rebuild (rsync -a mtimes), an unimplemented offline path, an overwritten pack, and now a pack-miss falling back to inline packing. Three are now loud (ORK_NO_STUB guard, offline-submit ENODEV, no-clobber abort). This one still is not — a read-mode miss on a weight the pack demonstrably contains should be an error, not a rebuild. That is the next hardening item.

A read-mode pack MISS on a weight the index contains is now fatal (ork_pack_miss_check, fork 28cafa703). The inline-pack fallback is correct for a genuinely cold weight — which is exactly why it is so dangerous otherwise: the run continues, builds the weight at a different tier than the pack specifies, and reports a plausible number.

Three outcomes, and the distinctions are the whole design:

Loader-scoped, not global. Each call site declares which dtypes it serves. A DT_I4 entry declining to load through the i4-native path is routing working, not a fault — a global check aborted a perfectly healthy i4a8 pack, which the negative control caught. Coverage: all five dtypes across four sites (i4-native dense, int8 dense, int4 expert, int8/i4a8 expert hot-slot). Not the GPTQ group path (a wcache miss, not a pack load) and not the hot-slot int8_only refusal (a deliberate policy decline to CPU). ORK_ALLOW_PACK_MISS=1 downgrades to a warning.

ork_i4a8_mm_load and ork_i8_mm_load have no offline branch (ork_i4_mm_load does). So on a no-device run every DT_I4, DT_I8 and DT_I8_ROT weight has been silently inline-packing — forever, not as a regression.

So the 5L claim that "unrotated i4a8 at -3.7% is the only surviving mixed-tier win" was itself measuring a fallback. The board's -3.3% for the same tier is real and points the same way; that is the number to quote.

It also retires a mystery: DT_I8_ROT at +33% was probably never a compute bug — those weights were not loading offline at all. The tier is unmeasured, not broken. And it explains why the last_abits fix changed nothing: it could not reach weights that never loaded.

The offline screen is only valid for pure DT_I4_NATIVE packs until the two offline branches exist. That is now enforced rather than remembered — an offline run of any other pack refuses instead of lying.

Next fix: offline branches in ork_i4a8_mm_load and ork_i8_mm_load, mirroring ork_i4_mm_load's. Then re-measure every mixed tier, on both paths.

ork_i8_mm_load and ork_i4a8_mm_load now have offline branches (ork-driver 51a302d), so a no-device run finally scores the PACK rather than an inline re-quantisation. examples/test_offline_load.c asserts CPU-dump → offline-load → memcmp of the codes, exact or fail, across a single tile, a K-slice, an Sn>1 wide-N slice and a non-pow2 K. In make test.

0.8B, VM offline, same machine, k=28 = attn_output + ffn_down, 256-token screen:

DT_I8_ROT was never broken. 5L called it "+33%, still broken" and 5M suspected it was merely unmeasured; 5M was right. It is the best tier available. Rotation also helps at i4a8 width (15.7034 vs 17.7138, -11.3%), which restores 5h's direction while replacing all of its magnitudes. The ordering is now monotonic in both rotation and precision for the first time in this document — which is what working code should produce, and is itself weak evidence the paths are finally correct.

Both are valid same-machine A/Bs, so 5f's cross-machine caveat does not explain this: one path is wrong. Prime suspect is the nibble→int8 inflate — the device uses orki_i4_expand_chan_f32, the offline branch sign-extends directly, and test_offline_load only proves the CPU encoder and the offline decoder agree. It never compares against the DEVICE decoder, which is a third implementation.

Next: a bit-exactness probe of device-inflate vs offline-inflate on identical bytes, the way test_i4_dump_cpu pins the tiler against the NPU oracle. Until it passes, quote i4a8 from neither path.

test_offline_load passed while the missing w->bscale bug was live, because it compares codes and scales are not codes. It caught the risk it was designed for (tile-layout arithmetic) and missed the completeness of the returned object. A round-trip test should assert the whole weight — codes, scales, shape — not just the part the author was worried about.

Every comparison in this document up to here was ork-tier-vs-ork-tier, against a "pure W4A4" control. That can rank our own tiers but it cannot detect a systematically poor baseline — it is how you optimise into a local minimum. So: the same 0.8B, the same machine, the same 256-token screen, run CPU-only (ORK_OFFLINE=rk3588 ORK_OFF=1 — ORK_OFF alone still initialises the backend, which fails on a device-less host and takes the context down with it).

The raw ratio (W4A4 / Q4_K_M = 1.32x) understates the problem and the "10x" intuition overstates it. The honest measure is excess PPL over the fp baseline, which isolates what quantisation costs:

Our W4A4 introduces ~3.4x the quantisation error of ggml's PLAINEST 4-bit format (+4.12 vs +1.22), and Q4_0 is the fair structural comparison — Q4_K_M (+0.25, so ~16x) uses super-block hierarchies and keeps selected tensors higher, which is not the same fight.

Note also that our best tier (rot-int8, +3.08) is not even pure 4-bit — it promotes 28 weights to int8 — and still loses to plain Q4_0 by 2.5x on excess error.

Q4_0 carries one scale per 32 weights. Our W4A4 carries one per channel — K = 1024..3584 weights. That is a 32-112x coarser quantisation grid, which comfortably accounts for a 3.4x error multiple without needing any other explanation.

Everything this session has pulled — activation clipping (-20.9%), weight clipping (-3.3%), GPTQ (-34% vs RTN), Hadamard rotation, tier promotion — has been optimising within a grid that is structurally ~32x too coarse. Those wins are real and they compound, but they cannot reach past the granularity ceiling. Clipping and rotation reshape the distribution presented to a quantiser; they do not add resolution to it.

They were deferred early in this session as "the expensive item", and the deferral was reasonable at the time because the only evidence was internal tier ranking. This is the first measurement that prices the cost of not doing it, and it is the largest single quality lever identified in the whole document.

The obstacle is performance, not correctness: the shipped grouped path measured 13.2x (M=1) and 283x (M=128) over per-channel — note that the earlier "1.18x" figure was retracted, it timed a hand-built proxy chain rather than the shipped path. But a performance problem with known mitigations is a different class of problem from a quality ceiling that no amount of clipping, rotation or error-feedback can lift.

Roadmap: Tier 16.

0.8B, VM offline, same machine, 256-token screen, all four cells built from the same source:

(excess over CPU bf16 = 11.7783 in brackets; CPU Q4_0 = 12.9964 = +1.22)

G=32 + GPTQ cuts quantisation error 38% for +24% disk (191.3 vs 153.9 MiB) and 2 minutes to build. Largest single quality lever measured in this document.

GPTQ and granularity are complements that overlap heavily. Each alone takes RTN's +14.50 down to ~+4.5 (-37 to -40%); together to +2.56 (-45%). Both attack error within a channel, so the gains do not add. The load-bearing detail: G=32 RTN (16.6911) is WORSE than per-channel GPTQ (15.9011) — a fine grid does NOT make GPTQ redundant, which was the specific question that arm was built to answer. Keep both.

5O concluded "per-group scales are the ceiling". Measured, Q4_0 is still 2.1x better on excess error (+1.22 vs +2.56). Granularity is the largest lever, not the whole story, and the roadmap entry is corrected.

Q4_0 is effectively W4A8: ggml quantises activations to Q8_0 (8-bit, per-block scales) for the dot product. Ours is W4A4. Section 2 established that the weight and activation error halves are at parity and independent, which predicts almost exactly the remaining 2x. That is PRIOR evidence, not a new measurement — and the distinction matters, because the direct test could not be run:

grouped scales + int8 activations is structurally unsupported. The grouped branch hardcodes 4-bit activation quant (s = mx/7.0f, clamp [-8,7]; ork_i4_scale_mse is likewise a 4-bit search) and dispatches to ork_i4_mm_run_grouped, which is int4-weights-only. A G=32 + rotated-i4a8 pack builds cleanly (191.3 MiB) and then refuses at run:

[ORK] blk.0.attn_qkv.weight: grouped W4A4 run REFUSED (K=1024 N=6144 G=32 M=64)

That refusal is correct — the code declines rather than emitting garbage. But it means the arm could never have measured W4A8 even had it run, and the container format compounds it: the compact i4a8 blob carries exactly N scales and cannot express groups at all, so only DT_I4_ROT_A8 (tiled int4 + scales in the pack index) can even represent grouped-i4a8.

Prerequisite: a grouped INT8 run path — per-group accumulation over int8-container weights with int8 activations. Roadmap Tier 16 item 2, now the critical path.

The first attempt at this experiment died on shell quoting (unmatched "), so no G=32 packs were built — and both arms then scored 26.2782, identical, because a nonexistent ORK_ORKPACK_PATH makes the run build an inline RTN pack and score that. Two matching numbers that look like a result.

The pack-miss guard (5M) does not cover this: it fires on a miss within a loaded pack, whereas here there was no pack at all, and self-healing an absent pack is deliberate behaviour. Always confirm SUCCESS: orkpack written before scoring, and prefer a script FILE piped to the host over nested ssh … "sh -c \"…\"" quoting — this is the fifth silent-plausible-wrong-answer of the session and the second caused by shell mechanics rather than by code.

Four hypotheses were tested against the W4A4-vs-Q4_0 gap before anyone measured the two quantisers on the same footing. Two of the four "findings" were my own measurement errors. tools/re/ork_qerr.cpp now does it directly — same tensors, same activations, fp64 reference, both schemes end to end.

Weight reconstruction (relative Frobenius, mean over 6 real tensors):

Matmul level — same activations (spiked Gaussian, 1% outliers x10; a clean Gaussian would flatter the unrotated schemes, since outliers are what rotation exists for), fp64 reference:

The chain is coherent and every lever appears in the right order and magnitude. Per-channel W4A4 starts at 6.4x Q4_0; grouping takes it to 2.2x, rotation to 1.56x, int8 activations to 1.07x. Rotation still helping at G=32 is independently confirmed here, matching the PPL result that refuted the antagonism idea.

Activation width is the dominant remaining lever for W4A4 — 0.14252 -> 0.09809, a 31% cut, larger than anything else outstanding. That vindicates the i4a8 / W4A8 direction despite its small PPL gain.

The 3.4x "chasm" of 5O was mostly per-channel granularity plus 4-bit activations. Fix both and the numerical deficit is 7%.

Comparing our matmul error (0.027, real activations) against Q4_0's weight error (0.0887) and concluding ours was better was invalid — different quantities. The dot product averages independent per-element errors, so both shrink; measured properly Q4_0's matmul error is 0.09144, not the ~0.01 that was speculated. Any claim resting on that comparison is void.

Note also the absolute values here do NOT match the real-activation ORK_MM_CHECK numbers (0.027 for W4A8): the synthetic distribution differs. Only the within-table ordering is meaningful.

Our best config is 1.07x Q4_0 at the matmul level but 1.84x on excess perplexity. A 7% numerical gap does not obviously produce an 84% quality gap. Candidate confounds, none yet tested:

• Q4_0 quantises token_embd; our run leaves it bf16 — we hold an advantage there and still lose.
• llama-quantize keeps output.weight at higher precision for Q4_0 mixes.
• Perplexity responds superlinearly to error, and is driven by worst-case tokens rather than mean error — the mean matmul error may simply be the wrong statistic.
• Per-matmul error does not compound with depth (measured flat, 0.008-0.041 across 24 layers, near-zero bias), so accumulation is ruled out.

A tail statistic, not a mean, is the next thing to measure: the worst-case per-token or per-channel error, since that is what perplexity punishes.

• tools/re/ork_qerr.cpp — weight and matmul reconstruction error, ours vs ggml, any GGUF. No NPU.
• ORK_MM_CHECK now fires on the grouped path too; it returned before the check, so it silently printed nothing for every grouped pack and looked like a probe with no findings.

27B could not be measured twice in a row. A freshly booted board loaded 400/400 weights and scored; after a few runs the same command died at ~40 weights, and only a reboot restored it. Three separate things turned out to be tangled together here, and two of the three "findings" along the way were measurement errors of mine.

1. rknpu_gem_object_destroy calls rknpu_iommu_domain_get_and_switch — the driver selects the object's domain on destroy itself (from /proc/<pid>/stack of a hung process).
2. That switch TIMES OUT: RKNPU: switch iommu domain time out, failed to switch iommu domain, id: 1.
3. WARN_ON(unmapped != size) fires in __iommu_dma_unmap ← iommu_dma_unmap_sg ← rknpu_gem_object_destroy — the unmap frees fewer bytes than asked.
4. Later allocations fail RKNPU: failed to allocate IOVA: -12 in rknpu_gem_get_pages, on a board with 12 GiB free and ork's own counter reporting the domain at 1085 MB of a 3900 MB ceiling.
5. Single-domain models (0.8B) never show it. Only multi-domain (27B) does.

Mechanism: multi-domain teardown needs a domain switch per object; under that churn a switch times out; the unmap then runs against the wrong domain; the IOVA is never reclaimed; it outlives the process and accumulates until reboot. Consistent with every observation, including why the 0.8B is immune.

The unmap-after-failed-switch is KERNEL-side. Userspace can reduce switch pressure. It cannot make that unmap correct.

Separate root cause, and the reason 27B overflows a domain at all. The sizer computed resident size from ctx->qbits — ONE GLOBAL PRECISION — so in a mixed pack every entry was counted at K*N/2, including DT_I4 and DT_I4_ROT_A8, which are int4 ON DISK but inflate to int8 containers at load. On the 27B ra8 pack that is 80 weights under-counted by ~3 GiB: domains planned for 11.6 GiB of an actual ~14.6 GiB.

Second under-count in the same loop: Bf gated at K<=4096 where the loaders build it for K<=10752, missing a full-K companion on every weight in between — and a 27B is full of K=5120.

Fixed by making the driver the single authority (ork_w_resident_bytes, ork-driver 587efff; consumer 95255a418): tile geometry, page padding and the Bf decision live in ONE place, parameterised by RESIDENT width. The pack-dtype → width mapping stays with the consumer, which is correct — pack format is its vocabulary, allocation is the driver's. Mixed 27B now plans 16 domains, was 12.

The general lesson: two models of the same allocation in two repos, with no gate to catch drift, WILL drift. They had drifted twice simultaneously and neither was visible as a code smell.

Believing MEM_DESTROY unmapped from whatever domain was active, ork_mm_free was changed to activate the weight's own domain first. It is wrong (fact 1 above: the kernel already does this) and it deadlocks: two switches race, the process enters uninterruptible D state, and timeout cannot kill it — observed 45 minutes against a 25-minute timeout, requiring a hard reboot. Reverted, with the reasoning left in ork_mm_free.

It did earn fact 2, which is what identifies the mechanism at all.

• dmesg | grep -c is not a cumulative counter. The ring buffer wraps, and verbose runs scroll old entries out — the count went 35 → 32 → 30 while leaking. This invalidated the "+5 per failed run" rate and the "cap fixed it → 0" result. Clear the ring (dmesg -C) before each run and count after.
• The control was the wrong model. "Clean exits leak nothing" was measured on the 0.8B, which is single-domain and therefore structurally incapable of showing a multi-domain teardown leak.
• IOVA capacity was blamed twice for what the driver's own counter (1085 MB of 3900) plainly showed it was not. 12 domains x ~4 GiB is ~48 GiB against a 12 GB pack; address space was never the constraint.

A fourth near-miss: ulimit -n = 1024 with every dma-buf import costing an fd looked like a clean explanation. It is a real ceiling worth raising (done, at init — grouped weights want K/G buffers per weight, ~45,000 descriptors at G=32) but it was NOT this bug: failures start at bcreate#42.

Section 5R concluded the leak was "kernel-side, not ours". That framing is now in doubt, and the reason is something 5R never considered: the board is not running a stock kernel.

uname -r                    6.1.115-vendor-rk35xx-sram     <- our build
CONFIG_ROCKCHIP_RKNPU_SRAM=y                               <- our config change
/boot/vmlinuz-...-sram.prepatch-bak                        <- and a SOURCE patch (Aug 11 rebuild over a Jul 8 image)
/boot/RESTORE_STOCK_KERNEL.sh                              <- reversible: two symlink flips

CONFIG_ROCKCHIP_RKNPU_SRAM makes the driver manage the 956 KB on-chip SRAM region and map it into the IOMMU domain. With MULTIPLE domains that mapping must be established or moved per domain — which lands squarely in rknpu_iommu_domain_get_and_switch, the exact function that times out (5R fact 2) and the exact function a caller-side switch deadlocked against.

It also explains the two things 5R could not:

• Single-domain models never leak. No switching, so no per-switch SRAM remap.
• Only multi-domain 27B degrades, and it degrades on the SWITCH, not on the allocation.

A stock driver that only ever mapped DRAM would have nothing to move on a domain switch.

Restoring stock is two symlink flips and is idempotent, with a documented microSD rescue path — so the test is cheap and reversible:

Either way it converts Tier 17's item 1 from an unmeasured hypothesis into a fact. The -sram run in flight is the control arm, so it costs one extra reboot, not a separate experiment.

We patched the kernel and did not write down what the patch was. .prepatch-bak (Jul 8) sits beside the current image (Aug 11) at the same byte size; a cmp will say whether the rebuild changed anything at all or was just a rebuild. Either way, a source modification to the driver under test, with no record, is a liability — it is a hidden variable in every board measurement in this document.

We never read the driver source. Every conclusion about rknpu_gem_object_destroy and rknpu_iommu_domain_get_and_switch in 5R was inferred from stack traces and dmesg, when the vendor source is public and the .239 VM has the toolchain. Pulling drivers/rknpu (queued) turns "the switch times out" into "here is the timeout, here is what it waits on, here is why SRAM changes it".

Lesson: when a driver-level bug resists four rounds of userspace hypotheses, check whether the driver is the one you think it is. This board has been running a modified kernel for the entire investigation, and that fact was in a memory note, not in the debugging.

After three failed attempts (a wrapping dmesg ring used as a cumulative counter, a single-domain control for a multi-domain bug, and a self-inflicted hang), one arm of the kernel bisect produced the number:

27B W4A4, clean boot, kernel 6.1.115-vendor-rk35xx-sram (#2 Aug 11)
run1: PPL = 10.6780   loaded=400/400   unmapWARN=30   iovaFAIL=0   switchTO=0

Protocol that made it trustworthy: dmesg -C immediately before the run, count after. Per-run, not cumulative, so a wrapping ring cannot corrupt it.

A fully SUCCESSFUL run leaks 30 mappings. All 400 weights loaded, perplexity was correct, nothing failed — and 30 unmaps still returned unmapped != size. The leak is not a failure path. It is the normal teardown of a multi-domain model.

And it happens with ZERO domain-switch timeouts (switchTO=0).

5R argued: teardown churn → a domain switch TIMES OUT → the unmap then runs against the wrong domain → WARN_ON(unmapped != size). That chain requires a switch timeout. Here there are 30 WARNs and 0 timeouts, so the timeout cannot be the cause.

The likely relationship is the reverse of what 5R assumed: the switch timeouts observed on degraded boards are a CONSEQUENCE of accumulated leakage, not its cause. Leaked IOVA accumulates, later switches have more to do or hit an exhausted allocator, and eventually they time out. First the leak, then the timeout.

That also explains a fact 5R had to hand-wave: the very first run on a freshly booted board leaks, even though nothing has yet gone wrong.

Why an unmap on a successful teardown frees fewer bytes than it mapped. Candidates, none tested:

• A size/alignment mismatch between map and unmap — the map rounds up (iova_align) and the unmap is handed a different size, so a fixed number of objects always under-unmap. The count being a round-ish 30 against 400 weights hints at a specific class of buffer rather than a random failure.
• Our own rknpu_job.c patch (see Kernel-Modifications) — an added rknpu_iommu_domain_put on job timeout. A double-put would underflow the refcount and free a domain early, so a later unmap addresses a domain that no longer holds the mapping. This is testable: the .prepatch-bak kernel on /boot is SRAM-without-the-patch.
• The SRAM enabler — the NPU's 956 KB SRAM is mapped into the domain and may unmap differently.

1. Is 30 constant? Same run twice on one boot. Constant ⇒ a fixed class of buffers. Growing ⇒ scales with something.
2. Does it scale with domains or with weights? ORK_DOMAINS_FORCE on a 155 MiB 0.8B pack (seconds per run) across 1/3/6 domains separates the two, and makes the whole question cheap to iterate.
3. int4 vs int8 at equal domain count — if int8 does not leak, the trigger is dtype-specific (the int4 per-row doorbell produces job timeouts; a clean int8 run does not), which would implicate our patch.
4. The three-kernel bisect — stock / SRAM-only / SRAM+patch, all three already built on /boot.

5R's "the destroy-path switch timing out is what corrupts the unmap" is withdrawn. The mechanism is unknown; what is established is that successful multi-domain teardown leaks, at a measured 30 mappings per 27B run, without any switch timeout.

ORK_DOMAINS_FORCE (a debug clamp-up added for exactly this) makes a 155 MiB 0.8B pack span N domains, so the question costs 15 seconds per arm instead of a 20-minute 27B run. Clean board, dmesg -C per run:

Single-domain is immune — 0 WARNs, confirming the structural asymmetry 5R inferred but never measured.

It does NOT scale with domain count. 2 domains and 6 domains both leak 30. So it is not "one leak per switch" or "one per domain".

It does NOT scale with weights. This pack has 102 weights / 155 MiB and leaks the SAME 30 as the 27B with 400 weights / 11.3 GiB (§5T). The leak is a fixed set of objects, and model size is irrelevant.

Correctness is unaffected — PPL is identical across every arm. The leak is silent in output, which is why it went unnoticed until the board started refusing allocations.

Almost certainly the per-domain SCRATCH, not the weights. Each domain lazily allocates its own ork_dom_scratch — regcmd, task, Af, Cc plus per-core mrc[], mtk[], maf[], mcc[] — which at 3 cores is ~16 buffers per domain; two domains ≈ 32. That fits 30 far better than any weight-derived count, and it explains the flatness: a small model only ever TOUCHES two or three domains regardless of how many are forced, so only those ever allocate scratch. (The 12-domain arm nudging to 35 is consistent with one more domain being touched.)

This also redirects the fix. Freeing WEIGHTS per-domain — what was attempted and reverted in ork_mm_free — aimed at the wrong objects. The scratch teardown is where a buffer belonging to domain N is freed while domain M is active.

Every 27B run spent studying this bug was wasted effort: the reproducer is a 15-second run on a 155 MiB model. The instinct to reproduce on the model that exhibited the bug (27B) rather than the smallest model that contains the mechanism (multi-domain, any size) cost hours, several reboots, one destroyed 17 GiB artifact and one hard wedge. Find the minimal reproducer before instrumenting the expensive one.

1. Confirm the count equals the per-domain scratch allocation count — instrument orki_bcreate/orki_bdestroy to log which buffers fail to unmap. Direct, and now cheap.
2. Free each domain's scratch with that domain ACTIVE at teardown (the correct version of the reverted ork_mm_free change, applied to scratch rather than weights).
3. The three-kernel bisect is now a 15-second-per-arm experiment rather than an hour.

With the 15-second reproducer (§5U), the bisect that would have cost an hour per arm costs a minute. 0.8B pack, ORK_DOMAINS_FORCE=3, two runs per kernel, dmesg -C before each:

Identical on all three, deterministic across all six runs. destroyFAIL=0, switchTO=0, PPL unchanged.

Neither of our kernel changes causes the leak. The SRAM enabler is exonerated. The rknpu_job.c domain-refcount put is exonerated — including the double-put theory, which was plausible enough to be worth testing and is simply wrong.

§5S is REFUTED. "The IOVA leak may be ours: the board runs a custom SRAM kernel" was a good hypothesis — it explained the single-vs-multi-domain asymmetry and pointed at exactly the right function — but it is not what is happening. The custom kernel remains a real hidden variable worth documenting (see Kernel-Modifications); it is just not this bug.

The leak is stock vendor rknpu behaviour: a fixed ~30 IOVA mappings per multi-domain run that the driver never reclaims, present on an unmodified 6.1.115-vendor-rk35xx.

The fix is not ours to make in the driver, but two things follow:

1. Userspace can avoid triggering it. The leak is per-RUN and fixed-size, so it is proportional to how many multi-domain processes are started, not to how much work each does. Long-lived processes leak far less than repeated short ones — which is an argument for the orkd daemon owning the NPU rather than each run re-establishing domains.
2. It is worth reporting upstream. drivers/accel/rocket is the mainline driver for this silicon; a teardown that drops ~30 mappings per multi-domain session is worth raising either way, and this is now a 15-second reproducer anyone can run.

Five mechanisms proposed, four refuted by measurement, and the survivor was found by making the experiment cheap rather than by reasoning harder:

The stack trace nobody had read settles it. Every WARN_ON(unmapped != size) comes from the kernel's own handle reap at fd close, on an imported object:

drm_release -> drm_file_free -> drm_gem_release -> idr_for_each
  -> rknpu_gem_object_destroy -> drm_prime_gem_destroy -> dma_buf_unmap_attachment
  -> __iommu_dma_unmap        WARNING: drivers/iommu/dma-iommu.c:691

Three driver facts combine:

1. rknpu_gem_prime_import_sg_table() never sets iommu_domain_id; rknpu_gem_init() uses kzalloc, so every import is filed under domain 0.
2. Its sg was mapped by dma_buf_map_attachment() during PRIME_FD_TO_HANDLE, against whichever domain was attached at that moment.
3. rknpu_gem_create_ioctl() does rknpu_gem_object_find(handle) and returns early on a hit — args->iommu_domain_id is silently ignored for imports. Userspace passes the domain; the kernel drops it, and the uABI offers no way to make it stick.

rknpu_gem_object_destroy() then switches to the recorded 0 and unmaps from the wrong page table. The freshly-allocated path (rknpu_gem_object_create()) switches first and then stamps — correct. Only import has the gap. Single-domain is immune because attached == recorded == 0, which is exactly why this never looked like anything but a multi-domain bug.

Fix: one line at import (kernel patch 03, Kernel Modifications). 30 WARNs → 0, and the board no longer dies on the third consecutive multi-domain run. PPL bit-identical.

Three of this investigation's conclusions were wrong, and each survived because it was never tested against the thing it claimed:

• "the switch times out, so the unmap runs against the wrong domain" (5R) — our own runs record switchTO=0 alongside 30 WARNs.
• "the leak is the per-domain SCRATCH" (5U) — the frames are drm_prime_gem_destroy, i.e. imports; and int8 parks 32 scratch buffers to int4's 16 while both leak exactly 30.
• "it fails on the int4 path" — int8 leaks identically (250 imports under non-zero domains).

Also learned: the board survived only two multi-domain runs per boot before MEM_CREATE returned EFAULT at ~176 MB of 3900. Several earlier readings in this document were degraded third-runs misread as results. Any pre-2026-08-24 multi-domain measurement taken as the 3rd+ run of a boot is suspect.

With the leak fixed, the 27B grouped pack finally runs. All three re-measured on the same build and kernel, W=64, clean boot + dropped caches per run, qwen36-27b-q8.gguf + stub-free full GGUF:

400/400 weights loaded in every case, 0 CREATE failures, 0 unmap WARNs. The two controls reproduce their pre-patch values exactly (10.6780 / 10.4421), which is what makes the comparison trustworthy — the kernel patch changes IOVA reclaim, not arithmetic.

Per-group scales carry to 27B and are the larger lever: 8.7%, against 2.2% for moving activations to 8 bits at per-channel granularity. This matches the 0.8B ordering (G=512 best at 13.2678).

The 176 soft resets on the per-channel rotated run are unexplained and worth a look; it produced the right number regardless.

The first attempt aborted at blk.48.attn_qkv.weight — and the abort was the pack-miss hardening working (before it, this silently repacked at the wrong tier and reported a confident 136986). The real cause was underneath:

CREATE FAIL: errno=22(Invalid argument) size=4096KB dom=16 dom_iova=4MB ceil=3900MB
[ork] domain 16 full — advancing residence to domain 17
RKNPU fdab0000.npu: RKNPU: invalid iommu domain id: 16, reuse domain id: 15

EINVAL on a domain holding 4 MB of 3900 is not exhaustion. RKNPU_MAX_IOMMU_DOMAIN_NUM is 16 — ids 0..15. The auto-sizer clamped to 63, the orkd owned_dom bitmask ceiling (a different limit), and sized domains at 1 GiB each on the premise that "many light domains are fine, count limit >> 8". 22.66 GiB asked for 24 domains; 0..15 filled, then each allocation past 15 returned EINVAL, which ork_domain_advance reads as "domain full" and answers by advancing to the next invalid id — leaking on every attempt.

Clamping the count also fixes the fill, since n_domains feeds domain_fill_cap: 22.66 GiB over 16 domains targets ~1.48 GiB each, well under the ~2.9 GiB edge the headroom clamp protects. Fork eef04ebc4.