Vendor-driver behaviour lives on its own page: Vendor-Kernel-Behaviour. This page is silicon/regcmd behaviour; that one is the shipped rknpu driver's own contracts and defects (IOMMU domains, imports, reset, refcounts). Several "hardware stalls" turned out to be entries there — in particular, job committed / task counter 0x0 / no interrupt has never once been silicon.

Hardware and rknpu-driver behaviours that are silent, order-dependent, or counter-intuitive — the kind that corrupt output or wedge the NPU without an obvious error, and that a fresh reader would otherwise re-derive the hard way. Each entry: the symptom, the root cause, and the guard ork-driver applies.

Scope note: perf-tuning levers live in the Optimization Roadmap; register/encoding detail lives in the regcmd ISA Reference; userspace safety layers in Robustness & Defensive Guards. This page is for "you must do X first or it breaks, and nothing tells you" traps.

The >4 GiB sliding window keeps weights resident across several 32-bit IOMMU domains (n_domains, dom_activate per submit — see NPU IOVA 4 GiB cap). Weights beyond domain 0 are imported zero-copy (dma-heap ALLOC → PRIME_FD_TO_HANDLE → MEM_CREATE) rather than native bcreate. Three separate constraints bite on this path — all root-caused 2026-07-06. Reproducer: tools/mc_import_probe.c.

⚠️ Validation trap first: verify both C[0] and C[last] (and ideally a full CPU-ref compare). The corruption here frequently leaves C[0] (first output column) correct while wrecking C[last]. A probe that checks only C[0] gives a false pass — this misled the original diagnosis (tools/chain_import_probe.c checked only C[0] and "passed" while output was actually corrupt).

• Symptom: an imported (dma-buf) weight in a non-0 domain produces non-deterministic wrong results — dropped K-partials, e.g. C[last]=14336 then 16896 then 13312 across runs, expected 18944. rc=0, no error, no wedge. Bad enough occasionally to wild-address → a 60 s NPU job timeout (job abort, ret: -22).
• NOT a multi-core bug (the tempting conclusion): forcing single-core still corrupts (core 0 alone: C[0]=18432 wrong). The per-core C[0]-ok / C[last]-wrong pattern is just non-deterministic noise. Native weights on the same non-0 domain are always correct.
• Root cause: the rknpu driver lazily initialises a domain's IOVA allocator / page table on its first buffer. If that first buffer is an imported dma-buf (SG-list), the import's pages land on wrong/aliased IOVAs → the NPU reads garbage for some weight tiles. A native allocation exercises the domain-setup path correctly; imports after that are bit-exact.
• Guard: ork_dom_prime(c, dom) — one tiny (64 KB) native bcreate per non-0 domain, before any import maps into it (no submit needed; alloc alone establishes the domain). dom_anchor[ORK_MAXDOM], once per domain (guarded), freed at teardown. Cost: 4 tiny allocs for a 7B, independent of chunking. Called at the top of ork_mm_load_i8_import.
• Upstream: looks like a genuine kernel first-buffer-domain-setup ordering bug (native-first works, import-first corrupts) — worth reporting / fixing in rknpu_iommu_dma_map_sg / the domain lazy-init; until then the anchor stays. Tracked as Optimization Roadmap Tier 10c.

• Symptom: a K-split submit that PC-chains many programs (e.g. ffn_down K=18944 → task_number=19), each referencing a separately-imported dma-buf mapping, is rejected: RKNPU_SUBMIT errno=110 (rc=-1, C=0). The CDMA chain-walker can't traverse that many foreign SG-list mappings. Native bcreate chains of the same length are fine; ~6 imported mappings works, ~19 faults.
• Guard: size-bounded chunked consolidation — pack a weight's tiles into a handful of imported chunks (tiles are chunk.dma+off views), so a chain walks few mappings. ork_mm_load_i8_import own_bufs[], ORK_IMPORT_CHUNK_MB (default 16 → down-proj ≈ 5 chunks).

• Symptom: importing a single dma-buf ≳ 16–64 MB fails at PRIME_FD_TO_HANDLE: Cannot allocate memory (16 MB OK, 64 MB ENOMEM; full 68 MB → WEIGHT NULL, an earlier variant hung the load R-state 571 s). The 32-bit domain's IOVA allocator can't find a contiguous range that large (fragmentation) — so it likely shrinks as a domain fills.
• Guard: the same chunking (Quirk 2) caps chunk size from above. 16 MB is a genuine sweet spot bounded on both sides — per-tile (many mappings) trips Quirk 2; 64 MB trips Quirk 3. Cannot consolidate a weight into one buffer.

Net: with the anchor (Q1) + 16 MB chunks (Q2+Q3), the zero-copy multi-domain import path is correct at full multi-core speed — validated mc_import_probe 5/5 bit-exact and 7B forced-import multi-core pp128 = 58.73 t/s, no wedge.

• Symptom: a submit that runs a standalone SDP activation op (act-LUT: exp/rsqrt/silu — ork_npu_exp_i16 etc.) immediately followed by a separate matmul submit on the same context reliably RKNPU_SUBMIT errno=110 (ETIMEDOUT) and trips the self-healing reset — per row-batch, which then discards the (correct) NPU activation output and falls back to CPU. The reverse order — matmul then activation — is fine (the on-NPU norm path does reduce-matmul → rsqrt-LUT with no stall; the fused-output SiLU rides the matmul's own SDP stage). Found building the composed softmax (ork_npu_softmax_f16: exp on NPU, then a Σ reduce-matmul).
• Root cause (open-source cross-reference): the RK3588 NPU is NVDLA-derived, and NVDLA's datapath is a fixed unidirectional pipeline — Convolution/MAC → SDP → PDP → CDP — with no backward path (NVDLA hwarch). Activation (SDP) sits downstream of the MAC. So "matmul→activation" is with the dataflow (and is exactly NVDLA fused mode, where the conv core streams into SDP via FIFOs); "activation→matmul" runs against it — SDP output would have to re-enter the conv core, which the hardware has no path for. Chaining a standalone-SDP submit into a fresh matmul submit leaves the pipeline mis-staged and the matmul never drains → timeout.
• Did the open stack note it / work around it? No — because it never issues this sequence. NVDLA/Rocket run activations as the fused output stage of a convolution (fused mode), or as a standalone memory-to-memory op in "independent mode" which "begins and ends with the assigned block performing memory-to-memory operations" (NVDLA hwarch) — i.e. a standalone SDP op lands its result in DRAM and is fully decoupled; it is never PC-chained backward into a matmul. The mainline accel/rocket driver + its Mesa userspace is convolution-centric and only resets the core at init (rocket_core_reset in rocket_core.c, called from rocket_core_init — not per-job), so it has no per-submit state-scrub either; it simply doesn't produce the offending order. There is thus no documented workaround to borrow — the architecture just doesn't do activation→matmul, and neither should we.
• Guard (ork-driver), interim: don't chain a standalone SDP activation into a matmul. Two safe shapes: (1) keep the activation on the NPU and do the following reduction/matmul on CPU — what ork_npu_softmax_f16 does (exp on NPU, Σ+normalize on CPU); (2) when a reduction must precede the activation, order it matmul→activation (the norm/rsqrt path).
• The REAL fix — PC-chain in ONE submit (what the vendor does), CONFIRMED by capture (2026-07-09): a full softmax capture (nn.Softmax, rknn-toolkit2) decodes to a 9-task PC-chained pipeline in a single submit — row-max → broadcast → x−max → load-exp-LUT → exp (SDP) → Σ = exp·ones (MATMUL) → 1/Σ → exp·(1/Σ) → output. Crucially the Σ is a matmul that runs right after the exp SDP op — the exact activation→matmul order that ETIMEDOUTs as two separate submits. It works because every task's 0101:0x0010 points to the next task's regcmd, so all 9 run within one submit — the pipeline never crosses a submit boundary / mode-switch re-warm. So the asymmetry is a per-submit-boundary effect, not a hardware "no backward path" law: SDP→MAC is fine when PC-chained in a single submit. ork_npu_softmax_f16 stalled only because it issued exp and the reduce as two submits. The forward reduction is NOT window-capped — validated on-NPU for reduction length N up to 1024 (refutes the CDP LRN n≤9 concern), and the geometry maps exactly onto attention scores [M,N] softmaxed over N (0x4058=N−1, 0x4030=min(M,128)−1). Full register map + dataflow: FWD_SOFTMAX_RE_WIP.md / regcmd ISA Reference.
• CONFIRMED on silicon (2026-07-10): ork_npu_replay_softmax_f16 runs the full vendor 9-task forward softmax on-NPU, bit-exact (uniform input → all outputs = 1/N). The correct mechanism is KERNEL-SEQUENCED per-task submits (one task_number=1 program per task, in dataflow order), NOT hardware PC-chaining — see the accel/rocket driver (one task per PC program, kernel re-arms on IRQ). Hardware-chaining attempts (concatenated regcmd + 0101:0x0010 + task_number=9) stalled at the chained LUT load; per-task submits (intermediates persisting in resident buffers, no reset between so the exp LUT survives) sail through. This is the empirical proof that the activation→matmul asymmetry is a per-submit-boundary/reset effect, resolved by keeping the tasks in one un-reset sequence. Interim ork_npu_softmax_f16 (exp on NPU + Σ on CPU) still applies until the geometry-parameterized replay is wired as a primitive.
• CORRECTION — the standalone-op cross-submit case has a concrete software root cause (2026-07-12, tools/mode_probe.c): the "standalone SDP activation → separate matmul submit ETIMEDOUTs" case is not the NVDLA "no backward path" dataflow law — it is a stale shared task descriptor left by the LUT-activation op. Proof from a 2-op transition sweep: ewmul_i16/ewmul_f16/add_f16 → matmul is SAFE, and exp_i16/silu_i16 → multi-core matmul is SAFE — both impossible if SDP genuinely could not feed the MAC. Only LUT activation (exp/silu) → single-core matmul wedged. See the dedicated quirk LUT-activation op poisons the shared task descriptor below and Exp-2026-07-12. (The single-submit PC-chain / kernel-sequenced replay above remains the right structure for real graphs like softmax for other reasons — LUT-load ping-pong, resident intermediates — but per-op interleave is now safe too.)

• Symptom: a standalone on-NPU LUT activation (ork_npu_exp_i16 / ork_npu_silu_i16 / gelu / rsqrt — all via act_lut_i16 → ork_npu_probe_silu_std_i16) followed — not necessarily immediately — by a single-core matmul (small N, e.g. the SSD CumBA bmm at N=16) RKNPU_SUBMIT errno=110 (task counter: 0x0 → soft reset). dmesg shows a real job timeout. The activation itself succeeds; the matmul wedges. ACT_RESET, ORK_NO_PINGPONG, and a mode-invalidate do NOT clear it (measured — tools/mode_probe.c E2/E3/E4) because it is not a hardware-pipeline state.
• This is why test_ssd_chunk_npu mode 2 wedged on CumBA "mid-sequence" while mode 1 (same exp→bmm) survived: mode 1's matmuls (N≥64) all run multi-core; mode 2 adds the CumBA (N=16), the only single-core matmul, which reads the poisoned descriptor.
• Root cause: the LUT op memsets the SHARED context task descriptor c->task to its own SDP program (regcfg_amount 1097 for the LUT-load, then 69 for the activation; enable_mask=0x18) and left it that way — it did not restore it. The single-core matmul path (run() → submit1) does not rebuild c->task; it relies on the init value (regcfg_amount=108, enable_mask=0xd, regcmd_addr=c->regcmd.dma, set once in ork_npu_init) persisting. So a later single-core matmul submits its 108-word matmul regcmd under the stale 69-reg / 0x18 SDP descriptor → the NPU reads a truncated/mis-typed program, dispatches no task (task counter 0x0) → timeout. It is a poisoned software descriptor, not HW state, so a hardware ACT_RESET cannot fix it.
• Why the trap is delayed & intermittent: multi-core matmuls use per-core c->mtk[] (rebuilt every call) and are immune; the plain 2-input SDP ops (ewmul/add) save+restore regcfg_amount/enable_mask around their submit. So c->task can stay poisoned across several ops (a multi-core matmul, then an ewmul that faithfully preserves the poisoned 69) until the next single-core matmul finally trips on it — which reads as "contextual/mid-sequence", not a clean A→B pair.

Transition matrix (RK3588, two ops as SEPARATE submits, warm context — tools/mode_probe.c):

• The int8 twin was missed for 6 weeks — fixed 2026-08-22. The save/restore below landed on the i16 LUT path only (npu/i16/act.c). ork_i8_npu_probe_silu_std (npu/i8/probe_sdp.c, behind every ork_i8_npu_silu/gelu/rsqrt/exp) kept poisoning c->task and returning. Symptom on the int8 side was a 2500 ms stall rather than an obvious wedge — mode_probe scored it ok(0,2500) = "safe", so the matrix never flagged it — but the stall reset-stormed and eventually Oops'd the kernel in rknpu_gem_object_destroy (Exp-2026-08-22). Fixed by the same save/restore (whole-struct, since the activation setup memsets the descriptor), on every exit including the LUT-load wedge path. GELU_I8 -> MM_F16 went 2500 ms -> 0.1 ms, and the full mode_probe sweep went from hanging-and-killing-the-board to 56 pairs in 48 s, 0 WEDGE. Measured proof it is not a mode-transition problem (fix ladder on the same pair): none = 2500 ms, invalidate = 5106 ms, reset = 5107 ms — a HW ACT_RESET makes it WORSE, because the cold re-warm is a 2-pass and each pass stalls. Do not reach for XSPEC/setdt here.
• Guard / fix (ork_npu_probe_silu_std_i16, and since 2026-08-22 ork_i8_npu_probe_silu_std): the LUT op saves c->task's regcfg_amount/enable_mask/regcmd_addr on entry and restores them on every exit — exactly the save/restore the ewmul/add ops already do. After the fix every transition above is SAFE with no interposed reset; test_ssd_chunk_npu mode 2 goes from a hard wedge to PASS (full-G1 rel-L2 5.8e-3, full-G2 5.9e-3). The minimal safe-sequencing rule for composing heterogeneous per-op NPU sequences is therefore just "every op must leave the shared c->task as it found it"; a fused single-submit chain is not required for correctness (it remains preferable for the submit-floor + LUT-ping-pong reasons above).
• Diagnostic knobs added alongside: ORK_MM_TIMEOUT (matmul-path submit timeout, mirrors ORK_EW_TIMEOUT; default 60000 — set low so a wedge-search self-terminates in ~2 s); ork_npu_mode_invalidate() / ork_npu_mode_reset() (the interposed-fix hooks the probe used to prove ACT_RESET does not help). See Exp-2026-07-12.

• Symptom: a multi-task hardware-chained submit (task_number>1, the HW walks 0101:0x0010→next) that contains a LUT-load task stalls non-deterministically at the LUT-load→consumer transition — the kernel job times out (errno=110, raw_status 0x80000000/0xc0000000 = a busy/hang state, NOT a maskable interrupt bit) with the task counter flaky (e.g. 4↔5 across identical runs). Fewer chained tasks can stall earlier than more — the giveaway that it's a race, not a deterministic wall.
• Root cause: submit flags bit RKNPU_JOB_PINGPONG (1<<2, i.e. the 0x4 in the common 0x5) enables double-buffered register banks (NVDLA PRODUCER/CONSUMER groups): the HW loads task N+1's register config into the alternate bank while task N runs, then swaps banks the instant task N's register config is done. That's a throughput win for register-config-only tasks (uniform matmuls) — but a LUT-load streams ~1000 entries into DPU LUT SRAM, a long-latency side effect NOT represented in the register bank. Ping-pong's fast swap advances to the consumer (e.g. the exp op) before the LUT SRAM has committed → the consumer reads a half-written LUT → hang/race.
• Guard: for any chain containing a LUT-load (or any task with an un-banked side effect), submit with ping-pong OFF (flags = RKNPU_JOB_PC = 0x1, not 0x5). The HW then fully completes each task — including the SRAM commit — before advancing. This is exactly what the vendor does: the captured softmax chain uses flags=0x1, while pure int8-matmul chains (run_chain_i8, flags=0x5) safely keep ping-pong on because their tasks are register-config-only. Validated: the full 9-task forward-softmax graph runs as ONE submit, bit-exact 9/9, ~471µs (vs ~2.5ms per-task) once ping-pong is off (ork_npu_replay_softmax_f16). Full recipe (all required): contiguous single-buffer image + single-delta rebase (so the LUT-load writes into the regcmd region as the vendor lays it out) + 0101:0x0014 next-amount =(n+3)/2 + ping-pong off. See regcmd ISA Reference / FWD_SOFTMAX_RE_WIP.md.

This is the root cause of what the section below calls the "int4 doorbell miss". That name is wrong — it named the first place the symptom was seen. The failure has nothing to do with int4, with the doorbell, or with NONBLOCK submission.

A job committed within ~1 ms of an iommu_attach_device() domain switch may fail to dispatch: accepted by the driver, PC completed-task counter stuck at 0x0, no interrupt, no IOMMU fault, MMU idle. Beyond ~1 ms after a switch, it does not happen.

Zero stalls outside the window across ~10,000 commits, on two paths that share nothing but the kernel submit code.

Tempting to read this as "the first N commits after a switch are poisoned". The int4 probe separates the two: 9004 commits sat at position 0-2 after a switch and only 5 stalled — and all 5 were in the sub-1 ms subset, while 8993 early-but->=1 ms commits were clean. Position is a proxy; elapsed time is the operative variable.

Inside the window the rate varies enormously by workload and shape — tn=2 is 75.7% in the 1.5B run and 0.2% in the int4 probe; within one run tn=4 is 90.9% while the larger tn=9 is 25.0%, so it is not monotonic in size either. Something else modulates it. The window is a necessary condition, and removing it removes every stall observed so far, but do not present it as the whole mechanism.

Each looked solid and each was a confound with switch-proximity:

1. "Domains >0 are stall-prone" (dom 2 at 42.9% vs dom 0 at 3.0%). Domain 2 is merely visited briefly, so most of its commits land just after a switch. Controlling for position: dom 0 = 0/180, dom 1 = 0/171, dom 2 = 0/20. The iova_reserve_iommu_regions asymmetry between domain 0 and the others is real but is NOT this bug.
2. "Stalls follow back-to-back commits (short inter-commit gap)". The first jobs into a new domain arrive in a burst, so short gaps co-occur with post-switch position. Enforcing minimum spacing between all commits gave no dose-response and inverted (500 us -> 12 stalls, 1500 us -> 46).
3. "ndom=2 triples the stall rate, therefore domain switching is expensive". Right observation, wrong reading: more switches means more exposure to the window.

dom_cooldown_us (rknpu, default 0) sleeps after a successful switch before the domain is released for use. It can usleep_range() rather than busy-wait because rknpu_iommu_switch_domain() is only reachable from process context, and it is charged once per SWITCH (~56 in a 1.5B run), not per commit.

Userspace already had the intuition and never quantified it: src/npu/core/domain.c:223 comments on a "first-submit-into-a-freshly-switched-domain hazard" and ork_dom_prime() exists for it.

Per-commit kernel profiling of BOTH populations. The stalled population alone was already known from userspace logs and was useless without the healthy base rate — every wrong turn above came from comparing a numerator against the wrong denominator. See "Measurement discipline" below.

SUPERSEDED — see the section above. The name is a misnomer: the same failure occurs on plain BLOCKING submits with no doorbell and no int4 involved. The cause is committing within ~1 ms of an IOMMU domain switch. The observations below about the failure's signature still hold; the framing and the suspected causes do not.

Status: software path fully exonerated with controls; cause still open (hung sub-block or silicon). The practical target is CHEAP RECOVERY, not prevention — see the protocol at the end.

On a multi-domain int4 (W4A4) prefill, some doorbell rounds never complete. Cost is time = 23.01 s + 3.61 s x misses (R2 0.967) — ~23% of 27B prefill. Numerics are never wrong (PPL bit-identical 10.6780 on every run at every setting): recovery always produces the correct answer, so this is purely a throughput defect.

It is a DISPATCH failure, not a stall. A kernel watchdog sampling the PC completed-task counter at 1 kHz reports, on every miss, all three cores at once:

core 0 NO TASK PROGRESS for 64 samples (64000 us), tasks=0/832 done (counter 0x0), cm=0x1 dom=1, job age 114993 us

The submit returns rc=0, the job is committed, and the hardware never executes a single task. Control (essential — a dead register reads identically): at an 8 ms threshold a HEALTHY ~30 ms job does NOT report — 3 reports across ~900 core-jobs, not hundreds. The counter genuinely advances on good work.

The output pattern misleads. The surface shows a contiguous written PREFIX ending on a 64-element boundary. That is NOT a partial write: cores get CONTIGUOUS op ranges (lo = i*S/nc, hi = (i+1)*S/nc), so a core that never runs leaves its contiguous slice at the sentinel. Nothing was written and then stopped.

Register map for the block (vendor rknpu_ioctl.h): 0x00 VERSION, 0x08 PC_OP_EN, 0x10 PC_DATA_ADDR, 0x14 PC_DATA_AMOUNT, 0x20 INT_MASK, 0x24 INT_CLEAR, 0x28 INT_STATUS, 0x2c INT_RAW_STATUS, 0x30 PC_TASK_CONTROL, 0x34 PC_DMA_BASE_ADDR, 0x3c PC_TASK_STATUS, 0xf008 ENABLE_MASK. ENABLE_MASK = 0 and INT_STATUS = 0 are NORMAL — they read the same on jobs that run. Do not chase either. INT_RAW_STATUS = 0 too, so the hardware is not signalling internally.

The failing shapes are not special, they are simply the BIGGEST: the 27B ffn_down is the largest, longest int4 op in the graph, which is why 12/12 model misses landed there.

The drain barrier does NOT transfer end-to-end: 5.8x on the probe, a WASH on the real 27B (8 vs 7 misses). The probe switches domains every rep so its rate is race-dominated; the model is residual-dominated. Weight any probe result against a model run before believing it transfers.

A miss costs ~3.6 s: a ~614 ms guessed detection window, then a device-wide ACT_RESET, then a resubmit. rknpu_soft_reset() asserts ALL SIX DT reset lines and msleep(100)s, and sets a global soft_reseting flag that makes rknpu_job_next() bail for every core — so recovering one core resets the whole NPU and destroys the other two cores' warm state.

The six lines are PER-CORE: reset-names = "srst_a0","srst_a1","srst_a2","srst_h0","srst_h1","srst_h2" (AXI + AHB per core; index N and N+3). So a targeted per-core reset is possible and is the cheap-recovery lever. RKNPU: soft reset, num: 6 prints num_srsts, the COUNT of DT reset phandles — it is a constant, not a reset counter. Count log lines instead.

1. Retire before switching domains (ORK_DOM_DRAIN, default on).
2. Bound programs per submit; prefer fewer, smaller submits over one huge one.
3. Batch work per domain to minimise switches.
4. Recover per-core, not device-wide, and detect from the task counter rather than a guessed timeout.

• netconsole WORKS and is the only thing that caught the hard-wedge panic. Persisted via /etc/systemd/system/netconsole.service (must be After=network-online.target; modules-load.d runs too early to bind eth0) + /usr/local/sbin/netconsole-up.sh. netpoll sends RAW frames so it needs a destination MAC — the gateway's MAC is the one that reaches the capture host here.
• ⚠ console_loglevel defaults to 4, so only messages BELOW level 4 reach the console and hence netconsole. <4>/KERN_WARNING test messages are silently dropped; this cost an hour of wrongly concluding netconsole was broken until a <3>/KERN_ERR message arrived instantly. kernel.printk = 8 4 1 7 is now persisted in /etc/sysctl.d/. Absence of low-priority lines in an old capture is NOT evidence the event did not happen — RKNPU: soft reset is LOG_INFO (level 6) and was invisible at loglevel 4.
• pstore/ramoops DOES NOT WORK on RK3588. Regions are reserved and re-registered every boot, but /sys/fs/pstore comes back empty even after a warm panic-reboot — the Rockchip restart re-initialises the DDR controller and wipes the region. Do not rely on it.
• /var/log is a 50 MB tmpfs (DietPi RAMlog, AUTO_SETUP_LOGGING_INDEX=-1) and journald retains no prior boots (journalctl -b -1 -> "No journal boot entry found") despite Storage=persistent.
• A durable fallback: a reader on /dev/kmsg writing /home/michael/ork-outbox/kern.log with fsync per message, on the NVMe, outside RAMlog.

Kernel log capture that actually survives: /var/lib/ork-logs/kmsg.log (systemd unit ork-kmsglog, dd if=/dev/kmsg ... oflag=dsync,append, on the NVMe rootfs). Two failure modes it avoids, both of which cost a dataset here:

• dmesg alone is not enough — the kernel ring OVERFLOWS on a bad run and silently truncates, which makes a coverage denominator read as zero.
• /var/log is tmpfs on this board (DietPi RAMlog, 50 MB), so anything written there is WIPED ON EVERY REBOOT and a verbose profiling run can fill it. An entire per-commit profile was lost to a reboot before it could be analysed.

The unit rotates the previous boot's capture to kmsg.log.prev on start, so the log from before a crash or reboot is still readable afterwards — which is the case that matters for wedges.

• A count that matches is not a cause. THREE false root causes this session came from exactly that: lost interrupts, marginal K geometry, and timeout_clean killing live jobs. The last one had treap == stalls == commit-done — a perfect match — and was only caught because the KILL AGE was logged alongside (157 ms, not the 1.5 ms the theory needed). Always log a second independent quantity — an age, a sequence, a checksum — next to any count that "matches".
• Always run the positive control in the same pass. "No mismatches" is worthless until the detector is shown to be able to fire; the regcmd checksum only became evidence once a deliberately corrupted word produced a mismatch.
• Match the control to the subject. Comparing a 1-task ping-pong-off job against a 2-task ping-pong-on stall produced two convincing false signals (task_ctrl readback, status(pre) 0x7000 vs 0x5000). Both vanished with matched jobs.
• Never sample a periodic workload on a modulo. The workload repeats a 4-job cycle (one real op plus three drain dummies); sampling every 500th completion aliased (500 %% 4 == 0) and captured ZERO matching jobs. Log the first N matches instead.
• Sample size. At a ~2% rate, n=400 gives ~10 events (Poisson +-32%) — direction only. Two wrong conclusions came from n=60 runs. Use >=4000 for any rate comparison, and bracket A-B-A: the absolute rate DRIFTS between sessions (the same config measured 5.25% and ~11% on different runs).
• Instrumentation perturbs. ORK_KMSG_SUBMIT (one extra syscall per submit) moved the stall rate 3.25% -> 6.00%. Use it to localise an event, never to quote a rate.
• Verify the knob fired. A pp=1 A/B returned a clean "no effect" from code that never executed (dyn.c early-returns for int4). It was caught only because removing HW chaining failed to produce the expected slowdown. Print a one-time confirmation from inside the instrumented path.
• Rebuild the right target. make i4_widek_stall_probe is a separate rule; a plain make leaves it stale, and a stale probe silently measures the old code.

ORK_RC_UNCACHED=1 (dropping RKNPU_MEM_CACHEABLE) makes the probe die with SIGBUS: the kernel then maps the buffer as Device memory, and ARM64 faults on UNALIGNED ACCESS TO DEVICE MEMORY, which the regcmd memcpy/NEON writes rely on. Not a viable way to test coherency.

• int8 mode entry needs an NPU reset. The first int8 submit wedges unless the NPU is reset when switching into int8 (fp16 cold-starts stale and the warmup handles it; int8 does not). run() issues RKNPU_ACT_RESET on the fp16→int8 transition, then re-warms. Mixing dtypes on one context works but each mode switch triggers a one-time re-warm. See src/npu.c run() and AGENTS.md §5.
• Never kill -9 an in-flight submit. SIGKILL mid-RKNPU_SUBMIT wedges the NPU/IOMMU (and can corrupt SPI on a hard power-cut), forcing a reset/power-cycle. Use SIGTERM + drain, and timeout on board NPU commands. See AGENTS.md §3.
• Weight-DMA amortization / M-tile ceiling. Single-core int8 is weight-DMA-bound; the M-tile cap is the 0x1040 schedule limit mg_max*64 (bit-exact), not an R-1/CBUF-resident-rows bound. See AGENTS.md "Weight-DMA amortization" and regcmd ISA Reference.
• Output zero-copy (ORK_ZC_OUT) is unsafe under concurrent multi-core. The per-core coherency bsyncs aren't serialized across cores. Off by default. See AGENTS.md and Optimization Roadmap Tier 7.
• 32-bit IOMMU = ~4 GiB per domain. rk_iommu v2 "40-bit" is physical PTE reach, not IOVA width. See NPU IOVA 4 GiB cap.
• Some int4-regcmd registers HARD-wedge the bus (kernel job-timeout can't recover; needs a power-cycle). Found while fuzzing the int4 multi-M scheduler (Exp-2026-07-07). Distinct from a soft/compute hang — a bad value mis-programs a DMA/partition so the bus locks and even sub.timeout won't return; only a power cycle recovers. Confirmed hazards: 0x1001/0x4010 and 0x1001/0x4100 (PPU/DMA output config — arbitrary overrides slip the DMA), and the 0x201/0x11xx CNA scratch region (e.g. 0x1104, though that one proved state-dependent — inert in isolation, wedged only cumulatively). These need follow-up RE (why do they bus-lock? are they the IOMMU/output-stride/CBUF-partition controls?). The fuzzer (tools/i4_multim_fuzz.c) blacklists them (i4_fuzz_blacklist.txt) and skips the 0x11xx range by default. See Exp-2026-07-07-INT4-Multi-M-Schedule-RE.
• Two submit-stress hard-wedge circumstances (2026-07-15), for follow-up RE. (1) Concurrent multi-context submits — two ork contexts' graph_computes on two threads with no serialization → IOMMU level 0 translation fault + soft reset (the single queue can't service two IOMMU contexts at once). (2) int4 large-M submit storm — the int4 1-submit-per-row × per-group path at M=512 (~8192 submits) → hard wedge (no ping, power-cycle only); isolated safe at M≤16 (≤256 submits). Distinct axes (concurrency vs count), both single-queue+IOMMU fragility. Guards + root-cause questions in Exp-2026-07-15-NPU-Hard-Wedge-Circumstances.
• fp16 M-scheduler has a valid Kp WINDOW [128, 2048), not just an upper bound. sched=1 programs the 0x1040 K-reduction schedule from scale=Kp/256; for small Kp it extrapolates too HIGH (K=64→0x1040=188, K=32→190) and miscomputes — constant-garbage output (rc=0, no wedge). K≥128 is fine (K=128→184), and the non-pow2 K=96 already took sched=0 (no 0x1040 override) so it worked, which masked the bug. This blocked fp16 attention at head_dim=32/64 (the common QKᵀ/AV dim) — surfaced by the strided ork_bmm_fp16 (i8 was exact at the same dims). Fix: gate fp16 sched=1 to Kp>=128 && Kp<2048; outside the window, sched=0 (general path) is correct. Validated test_bmm K-sweep 32…256. (The upper bound is the pre-existing ">8 rows garbage at Kp≥2048" gate.) See src/npu.c run()/synth().
  • Confirmed + extended 2026-07-12 (Exp-2026-07-12-Matmul-Shape-Boundaries): the single load-bearing register is 0x201:0x1040 — a silicon register-bisect (ork_f16_fuzz_add(0x201,0x1040,·)) shows overriding only 0x1040 0xBC→0xB1 (template default) at M=64,K=64 flips zero-output (rel-L2 1.0) → bit-exact (3.8e-8). The bug re-appeared in ork_bmm_fp16_fused (the on-NPU SSD-scan chain) and the raw ork_*_probe_*mm helpers, which all still hardcoded sched=1 — same gate now applied there (sched=(K&(K-1))==0&&K>=128&&K<2048), validated bit-exact across the scan shapes (K∈{32,64,96,128}, N∈{64,128}). int8 twin: synth_i8's formula scales by K/512 (vs fp16 K/256), so its invalid band is wider — K<256 (K∈{32,64,96,128} all zero; K≥256 correct); production int8 is safe because the run path only sets sched=1 at K∈{512,1024}, but any raw-synth_i8 chain must use sched=0 below K=256. Boundary is N- and M-independent.
• int4 native multi-M is CBUF-limited batch mode. Reg 0x201/0x107c = K/16 enables a resident-activation batch: rows/submit = min(M/2, 16384/K) (activation budget) AND the per-core N-slice weight must fit Ncore*K <= 131072 int4 (weight budget) — exceed either and far rows/blocks silently don't compute. int8 avoids this by streaming activations (its 0x1040 schedule, which corrupts int4). int4 streaming is unconfirmed (open RE). See the RE Roadmap "int4 STREAMING mode".
• Concurrent per-core NONBLOCK doorbell submits intermittently DROP (~1/2000–4000; accepted-but-never-dispatched). (2026-07-19) When an int8 matmul rides the multi-core NONBLOCK doorbell — ork_dyn_begin_mc (S>1 streams: QKV, gate/up) or ork_dyn_begin_colsplit (M=1 decode, N-columns split across cores) — a round occasionally is accepted (every per-core RKNPU_SUBMIT returns rc=0) yet the HW never executes it: no output sentinel ever lands, no task-completion IRQ fires. Timing-, gap-, ping-pong- and interleave-independent (all measured — not a retirement/EBUSY/mid-round-reset race; those were disproven dead-ends). Rate rises with submit density. Strongly consistent with a lost doorbell / IRQ-rearm race in the stock rknpu driver's concurrent per-core dispatch (kernel-side; unprovable from userspace because RKNPU_GET_TOTAL_RW_AMOUNT and task int_status read 0-always on this kernel — the output sentinel is the ONLY reliable completion signal; do not trust those counters).
  • Deceptive colsplit signature: a dropped core's N-column slice retains the previous iteration's output (fixed test inputs → still correct) except its re-seeded sentinel column, so a whole-slice drop looks like a single wrong interior column. It's the per-core sentinel, not an interior-column compute bug — a validator that checks only C[0]/C[last] can miss it entirely.
  • Why it hid so long: introduced by the blocking→NONBLOCK-doorbell spine migration (prior blocking submits waited in-kernel — no dispatch race to lose). Below single-run detection: make test runs the one guard (test_stream_interleave) ~420 rounds → passes ~80 % of the time → looked reliable; only surfaced under dozens of back-to-back runs. CI can't run the NPU tests, so no automated repetition accumulated.
  • Fix (ork_dyn_end + mc_recover_resubmit, src/npu.c): detect via a bounded sentinel timeout (300 ms for mc int8 — real decode/stream lands ≪ that), then RKNPU_ACT_RESET + ~1 ms settle (a resubmit into a not-yet-quiesced NPU re-drops → sticky/correlated miss) + re-clean + re-seed + resubmit, up to 6 retries. Both ORK_MC_ROUND and colsplit stash their round context in the chain handle so ork_dyn_end can replay it. Fires only on a real drop (≈0 common-case cost). test_stream_interleave went 25/40 → 40/40; tools/mc_miss_repro (stop-on-first-miss stress loop) survives 12000 iters bit-exact with recoveries logged (ORK_MC_DIAG=1). submit1_db (single-core nc==1) is not yet covered — no op was observed dropping on it (M=1 int8 decode routes through colsplit, not submit1_db); add the same pattern if one ever does.
  • Lessons: (1) a self-validating test run once per gate cannot catch a sub-0.1 %/round drop — use a stop-on-first-miss stress loop; (2) verify a "counter" actually advances on this kernel before trusting it — dma_rw=0-always was a trap that produced a false "never dispatched" reading and, when used to gate recovery, a reset storm that hard-wedged the board; (3) moving a path blocking→NONBLOCK trades latency for exposure to this dispatch race, so the recover is mandatory on every doorbell path.

• Symptom (delayed, deadly): after an accepted-but-never-dispatched doorbell drop (the quirk directly above), if recovery abandons the job's output buffer — frees it, or de-escalates to a different code path that reuses a fresh buffer instead of completing the dropped job — the box looks fine for ~1–2 minutes, then the kernel dies: BUG: Bad page cache in process kworker … pfn:… (a run of contiguous PFNs) → page dumped because: still mapped when deleted on the shmem eviction path (filemap_unaccount_folio ← truncate_inode_folio ← shmem_undo_range ← shmem_evict_inode ← evict ← iput ← __dentry_kill ← dput) → Unable to handle kernel NULL pointer dereference → Oops in rknpu_gem_sync_ioctl (i.e. our own bsync) → soft-reset cascade → hard wedge (power-cycle only). The delay makes it easy to mis-blame the next thing that runs.
• Root cause: an NPU job that was accepted but never dispatched (nonblock doorbell drop; hw_elapse=0, int_status=0, iommu=1 — the NPU is healthy, it just never ran) still holds references to its output buffer's shmem/DMA-buf pages. Those refs are released only when the job COMPLETES — either it finally dispatches and lands, or the kernel's ~60 s watchdog reaps it. RKNPU_ACT_RESET alone does NOT release them (verified: a reset-then-abandon still crashed at teardown). So destroying that buffer (context free / bdestroy, or the process exiting) while the phantom job still references it = pages "still mapped when deleted" = the crash.
• Rule: never free or hand back a DMA buffer while any in-flight OR dropped NPU job may still reference it. On a doorbell-drop recovery you MUST get the job accounted first: (a) complete it on the original buffers — resubmit → poll to landing, as int8 mc_recover_resubmit does; the colsplit/mc per-core buffers are context-level and reused, so the completing job reaps the phantom refs — or (b) let the kernel watchdog reap it before freeing. Do NOT de-escalate to a path that leaves the colsplit buffers behind with the drop still outstanding.
• Why int8 nonblock is safe and the fp16-colsplit sentinel path was not: int8's recover always lands/completes on the reused buffers → refs released → clean teardown. The fp16 sentinel recovery detected the drop fast (800 ms nonblock poll) but then de-escalated to nc=1, abandoning the colsplit buffers with the dropped slice-1 job never completed → the delayed crash. Fix direction: complete-the-job (resubmit to landing), never abandon.
• Diagnostic that nailed it: ork_npu_dump_state / ork_dyn_dump fired at detection, before recovery, with fflush + live-streamed over ssh (not captured in a shell $(…), which buffers until exit and is lost when the box hangs). Shows the drop is always the 2nd K-slice (Kslice=1), hw_elapse=0. Fire your post-mortem before you touch the NPU, and stream it off-box — a hard wedge loses anything still in a local buffer. See COLSPLIT_MULTIPREC_WIP.md "ROOT CAUSE FOUND via auto-dump (2026-08-05)".

The 2026-08-05 crash surfaced in rknpu_gem_sync_ioctl. The identical root cause also lands in the destroy path, which is worth knowing because the trace looks unrelated:

Unable to handle kernel paging request at virtual address 0000002710000008
Internal error: Oops [#1] SMP   Comm: mode_probe   pc: virt_to_folio  lr: kfree
  virt_to_folio <- sg_kfree <- __sg_free_table <- sg_free_table
  <- rknpu_gem_object_destroy <- rknpu_gem_free_object <- drm_gem_object_free
  <- drm_gem_handle_delete <- rknpu_gem_destroy_ioctl <- drm_ioctl

Tell-tale: x20 = 000009c400000007 — 0x9c4 = 2500 = the ORK_MM_TIMEOUT that run was started with, packed against a 7. Scalar data where the sgl pointer belongs: the table was freed and its memory reused, then walked again at destroy.

Trigger (mode_probe, needs a SEQUENCE — neither pair alone does it): GELU_I8 -> MM_F16 completes but burns the full timeout and emits ~14 back-to-back RKNPU: soft reset; the next transition's buffer realloc then issues the GEM destroy into that state.

A post-reset SETTLE DELAY DOES NOT HELP — measured, don't retry it. A settle window inserted before MEM_DESTROY was tried at 2 ms and at 100 ms: identical Oops both times, same trace, and the 100 ms run took the board down. That is exactly what the rule above predicts — the refs are released when the job completes (or the ~60 s watchdog reaps it), not after any amount of wall time. The mitigation was reverted rather than shipped, because a delay that looks protective and isn't is worse than nothing.

So the rule stands unchanged and applies to destroy as well as sync: complete the job on the original buffers, or let the watchdog reap it — never just wait, and never abandon.

Note this raises the value of the ork_dyn_progress frontier fix (2026-08-22, dyn_ctl.c): before it, a hole at a non-final task read as success, so the recover loop never fired and the dropped job was silently abandoned — precisely the pattern this rule forbids. With recover reachable again, those jobs now complete on their original buffers and release their refs.

Aftermath if you hit it: Oops [#1] only taints the kernel. The box can survive minutes and then die on the next activity — once on a purely read-only command — so the death gets mis-blamed on whatever ran next. Full write-up + kernel-log capture recipe: Exp-2026-08-22 mode_probe Kernel Oops.

This resolves the fp16-colsplit drop above. The fix is PREVENTION, not recovery — and the two failure classes must not be conflated:

• The fp16 drop is NOT the benign int8 dispatch-miss. It is a wild DMA: when the 3 cores loop their K-slices, the NPU prefetches the next slice's weight base (Bb[ks+1]) while Bb[ks]'s fp16 weight-DMA is still draining → two distinct dma-buf fds on the CDMA at once → mis-addressed fetch. fp16's 2-byte weights double the fetch bytes; int8 tolerates the same pattern, fp16 wilds. Signature: the drop is ALWAYS the boundary slice (Kslice=1 at Sk=2, never Kslice=0) — textbook cross-boundary prefetch. Same-buffer concurrency (3 cores reading regions of ONE buffer) is benign; only cross-buffer (distinct fds) wilds. A CPU lockstep barrier cannot fix it — it can't gate a hardware prefetch.
• A real fp16 drop is UNRECOVERABLE from userspace (exhaustively tried 2026-08-05: in-place resubmit → sticky re-drop + uninterruptible D-state hang; close(fd) → refcount underflow; timeout_clean reap + fast-detect + reset-first → kernel panic). The wild write corrupts kernel RAM at the drop instant, microseconds before any 800 ms / 24 ms detect can react. And any BLOCKING submit that can't dispatch into the post-drop NPU (e.g. an nc=1 fallback) enters the kernel's 60→120→180 s continue_wait → rknpu_job_abort, which does rknpu_iommu_domain_put() → msleep(100) → rknpu_soft_reset() — tears down the IOMMU 100 ms BEFORE halting DMA. The device-global iommu_domain_refcount gates domain switching (switch only if refcount==0); a dropped job's double-cleanup over-decrements it → a later switch iommu_detach_device under a live matmul → wild write → slab corruption → Oops/panic in unrelated processes (percpu_ref_put, rb_insert_color, shmem eviction), delayed 1–2 min. rknpu_job_timeout_clean is safe only because it resets BEFORE teardown; abort is not.
• THE FIX = CONTIG (w->Bbc, default-on for Sn==1): concatenate all K-slices into ONE weight buffer → there is no dma-buf boundary, so the "next base" is just the next address in the same fd/mapping → the prefetch stays in one mapping → no wild → no drop → recovery never fires. 3 cores read regions of the one buffer concurrently (benign). Validated: 1000-iter @256×3584×3584 = 0 drops / 0 wrong / 0 panic (the per-slice path dropped on iter 0), full shape suite BIT-EXACT (Sk=1 and Sk=2), make test all-PASS, 2.6× preserved (nc3/nc1=0.38×).
• The one-line blind spot that hid CONTIG: is_valid_dma_addr (src/npu.c) validated the weight base against w->Bb[]/w->Bf[]/scratch but not w->Bbc → a valid Bbc.dma+offset base was FALSE-flagged "wild/unallocated (outside all valid buffers)" → validate_regcmd refused → CONTIG fell back to the wedging per-slice path. CONTIG's address math was correct the whole time. Lesson: when a "wild address" is inside an expected buffer's range, suspect the validator's known-buffer list before the address arithmetic.
• Taxonomy to keep straight: benign accepted-but-never-dispatched miss (no DMA fired — int8/int4 doorbell drops) is recoverable (mc_recover_resubmit, reset+reap+resubmit-to-landing, proven 12000 iters); wild-DMA drop (fp16 cross-buffer) is not — it must be prevented. The recovery protocol (adaptive sentinel detect, reap_stuck clean timeout_clean reap, ork_dummy_probe int8 health-gate, reset-first) is retained as the safety net for the benign class and any future Sn>1 fp16 shape where CONTIG's single-buffer assumption doesn't hold. Gate/knobs: ORK_F16_COLSPLIT (path), ORK_F16_NO_CONTIG (A/B opt-out), ORK_F16_TRACE=<file> (tailable step log), ORK_CAMPAIGN (per-iter bit-exact).

RE'ing rkllm's fast int8 "M-fold" prefill matmul surfaced several traps. The full fold data layout (bit-exact validated on RK3588 via tools/re/validate_layout.c):

• A (input) nc16 = (k/16)*(M*16)+m*16+(k%16) — NC1HWC2 C2=16, M folded into WIDTH.
• W (weight) woff = ((n/32)*(K/32)+(k/32))*1024+(n%32)*32+(k%32) — the documented RK3588 int8 rknn_matmul NATIVE layout (N/32,K/32,32,32) (from rknn_matmul_api.h; per-chip (subN,subK) table in [Reverse-Engineering-Roadmap] "Porting the fold to other Rockchip NPUs"). Do not board-RE the weight pack — read the header.
• C (output) c4 = (n/4)*(M*4)+m*4+(n%4) — NC1HWC2 C2=4, int32.

Gotchas:

• ORK_REPLAY_RESET is a WEDGE TRIGGER, not a stabilizer. It issues RKNPU_ACT_RESET before the submit; on the mfold replay path that reset hard-wedges the board (needs a power-cycle). WITHOUT it the replay runs clean and repeatable (validated bit-exact twice). Do NOT set it for mfold replay.

• The captured output mm_C.bin is STALE — don't trust it as a bit-exact reference. rkllm reuses a tiny pool of ~5 scratch C buffers (e.g. 0xb17a000), and the rknpu_dump.so shim dumps C after the whole run, so it holds a LATER submit's result, not the captured tile's. The captured (A,W,C) is NOT a consistent triple. Validate with a SELF-CONSISTENT test instead (ork's own A/W → replay → compare CPU ref), not against the dumped C. (The int32 DST output IS raw signed A·W: no requant / zero-point — confirmed by one-hot injection.)

• Injection-ramp aliasing when decoding a layout probe. A recoverable ramp A[i]=(i%251)-125 aliases a value to i ∈ {v+125, v+125+251, …}; e.g. val −120 → i ∈ {5, 256, …}. Pick the i that is a valid nc16 index (256 = nc16(0,32) → k=32), not the smallest. A naïve inverse mis-reads the k it maps to.

• rkllm runs the fold as a task_number=21 weight-resident chain, 3-core round-robin — the per-tile speed win is weight residency across the chain, NOT a single fold tile (a lone width-8 tile does not amortize the weight DMA and loses ~4× to ork's normal M-scheduler at M=228: ~17 ms vs 4.6 ms).

• 0x40c0 (DPU_SURFACE_ADD) is an elem-size CONFIG, NOT an output IOVA — preserve it, do not patch it. 0x20 int8 / 0x80 int32 / 0x400 M-fold (=128*M@M=8). Patching it as an address points the WDMA at an unmapped low IOVA and hard-wedges. (An earlier note in Exp-2026-07-28 called it an IOVA — corrected there.)

• M>8 mfold runs as a chain of M<=8 tasks, and it is bit-exact (2026-07-30). ork_npu_mfold_chain_cap chains P copies of the captured bit-exact width-8 tile (mm_regcmd_m8.txt) in one task_number=P submit, shared resident weight, PC-chained at 216-219: 0-mismatch at M=8/16/32/64, no wedge. Required regcmd prep: preserve 0x40c0 (above) and neutralize the captured chain descriptor keeping the reg-ids (zero only the value halves of the 0x0010/0x0014 block-0x101 entries, like ork_npu_replay_i8) — zeroing whole words 216-219 drops the 0x0014 terminator and even one task hangs. Perf caveat: each tile re-streams the full weight (0x1040=0xb1) -> linear, ~7x slower than ork-normal at M=228; the 1.6x needs weight residency (load-once/reuse) across tiles.