RESOLVED 2026-08-22 — see "The fix" at the bottom. mode_probe is usable again (full sweep: 56 pairs, 48 s, 0 WEDGE). The kernel Oops is still a genuine rknpu driver defect, but ork-driver was provoking it, and that provocation is now removed.

Original outcome: root-caused, and it is a KERNEL DRIVER bug, not an ork-driver correctness bug. mode_probe triggers an Oops inside rknpu_gem_object_destroy → sg_free_table, which taints the kernel; the box then dies on the next activity. Until this is addressed, mode_probe cannot be used as the validation gate that AGENTS' mode-transition protocol calls for.

mode_probe stalls partway through the transition matrix. The wrapping timeout does not reap it (the process is stuck in an uninterruptible ioctl and ends <defunct>), and the board frequently goes fully unreachable — no ping, not just SSH, which is a kernel-level hang rather than the usual NPU wedge. Three power cycles via the Rock 5B Plug were needed during this investigation.

Captured by streaming dmesg -w to an append-only file with a 1 s sync loop (a plain dmesg > file loop did not survive the power cut, and Storage=persistent journalling did not retain the crashed boot either):

RKNPU: soft reset, num: 6          <- x14, back to back
Unable to handle kernel paging request at virtual address 0000002710000008
Internal error: Oops: 0000000096000004 [#1] SMP
CPU: 7 PID: 2369 Comm: mode_probe  Not tainted 6.1.115-vendor-rk35xx-sram #2
Hardware name: Radxa ROCK 5B (DT)
pc : virt_to_folio+0x18/0x2c
lr : kfree+0x94/0xe8
x20: 000009c400000007
Call trace:
  virt_to_folio
  sg_kfree
  __sg_free_table
  sg_free_table
  rknpu_gem_object_destroy+0x134/0x188
  rknpu_gem_free_object+0x34/0x4c
  drm_gem_object_free
  drm_gem_object_put
  drm_gem_object_handle_put_unlocked
  drm_gem_object_release_handle
  drm_gem_handle_delete
  rknpu_gem_destroy_ioctl+0x9c/0xb4
  __rknpu_gem_destroy_ioctl
  drm_ioctl_kernel / drm_ioctl / vfs_ioctl / __arm64_sys_ioctl

So: userspace calls the GEM-destroy ioctl to free a DMA buffer, and the driver faults walking that object's scatter-gather table — sg_free_table calls kfree on a pointer that is not one.

x20 = 000009c400000007 is the tell: 0x9c4 = 2500, the ORK_MM_TIMEOUT this run was started with, packed against a 7. That is scalar data sitting where a pointer belongs — the sgl allocation has been freed and reused, i.e. a use-after-free / double-free in the driver's GEM teardown.

Bisected with mode_probe <a> <b> (single pair) against mode_probe <a> (one A, every B):

Neither pair reproduces alone. The chain is:

1. GELU_I8 → MM_F16 completes but the matmul burns the full 2500 ms timeout (ok(0,2500)) — the known mm-after-SDP behaviour — and emits a burst of ~14 RKNPU: soft reset.
2. The next transition needs a buffer realloc (AGENTS: "Moving between modes may need an RKNPU_ACT_RESET, a cold 2-pass re-warm, and/or a buffer realloc"), so ork-driver issues a GEM destroy.
3. The driver's bookkeeping for that object did not survive the reset storm → Oops in sg_free_table.

Note ork_npu_mode_reset itself is only ACT_RESET + clearing software warm flags; it frees nothing. The destroy comes from the transition's realloc.

4. Oops [#1] taints the kernel. The box does not necessarily die at once — in one run it survived long enough to read the log — but it collapses on the next activity. It went down once from a purely read-only command issued after the Oops, which is why the failures looked erratic and were initially mistaken for something the workload was doing.

Kernel (rknpu DRM driver). Freeing a GEM object must not oops regardless of what the NPU did beforehand; a corrupted sg_table after a reset is a driver lifecycle defect. It is the same root cause as the 2026-08-05 quirk (a never-completed job still holding refs to its buffer's pages), surfacing in the destroy path rather than sync — see NPU-Quirks. ork-driver's part is only that it provokes it, by destroying GEM buffers immediately after a reset storm.

This also explains a long-standing observation: "wedges" that leave SSH alive are NPU-level, but these full-network deaths are this Oops. They are different failure classes and should not be lumped together.

1. Do not treat mode_probe as a required gate until this is resolved — it is a board-killer, not a validator. Drain/mode changes currently have to lean on make test across the keep-warm knobs plus the ORK_DEBUG_RESET count diff. AGENTS' mode-transition protocol should say so.
2. Userspace mitigation worth trying: quiesce before destroying GEM buffers after a reset. TRIED AND DISPROVEN (2026-08-22). A settle window before MEM_DESTROY was measured at 2 ms and at 100 ms — identical Oops both times, same trace, and the 100 ms run took the board down. Reverted rather than shipped. This is a re-discovery: NPU-Quirks already records (2026-08-05) that "RKNPU_ACT_RESET alone does NOT release them ... refs are released only when the job COMPLETES" — so no delay could ever have worked. The standing rule is complete-the-job (resubmit to landing) or let the ~60 s watchdog reap it. Read NPU-Quirks before attempting a mitigation here.
3. Remove the upstream cause: the reset storm comes from the mm-after-SDP transition burning the full timeout. That is the Phase-2 XSPEC[XP_SDP].setdt item (AGENTS §Mode-transition layer); note the earlier attempt recorded it as "SAFE but INSUFFICIENT", and this Oops is a second, independent reason to revisit it.
4. Cross-check accel/rocket (AGENTS §open-source reference) for how the mainline driver sequences GEM teardown against a reset — it is the authoritative comparison for this exact path.

A hard power cut loses ordinary logs. What worked:

sudo sh -c "nohup dmesg -w >> /var/log/knl.log 2>&1 &"      # append-only, not truncate-rewrite
sudo sh -c "nohup sh -c 'while true; do sync; sleep 1; done' >/dev/null 2>&1 &"
sudo sh -c "echo '=== REPRO: <what> ===' > /dev/kmsg"        # marker to find it again

dmesg > file in a loop and Storage=persistent journalling both lost the crashed boot.

The reset storm that precedes the Oops came from the SDP -> single-core matmul stall, and that stall was ork-driver's own bug: the int8 LUT path (ork_i8_npu_probe_silu_std) left the shared c->task descriptor poisoned at regcfg_amount=69 / enable_mask=0x18. The i16 twin had had the save/restore since Exp-2026-07-12; the int8 one never got it. See NPU-Quirks.

Measured proof it was never a mode-transition problem — the whole XSPEC/setdt ladder on GELU_I8 -> MM_F16:

A full HW ACT_RESET makes it worse (the cold re-warm is a 2-pass, each pass stalls). That is the signature of a poisoned software descriptor, exactly as the quirk says.

After adding the save/restore:

make test passes at default and under both keep-warm knobs, with the reset count unchanged at 51.

What is still a kernel bug: freeing a GEM object must not Oops regardless of what the NPU did beforehand. We no longer walk into it, but the driver defect is unfixed — so the NPU-Quirks rule (never free a buffer a dropped job still references; complete-the-job or let the watchdog reap it) still stands.

Why it hid so long: on the int8 side the poisoned descriptor produced a slow success, not an error. mode_probe scored it ok(0,2500) and called it "safe", so the transition matrix never flagged it — the 2500 ms in that cell WAS the bug, in plain sight, for six weeks.