Exp 2026 10 09 PR390 Alone Does Not Fix The Domain Switch - oRKLLM/ork-driver GitHub Wiki

Exp-2026-10-09 — PR #390 alone does not fix the domain switch; the missing piece is reclaim-and-retry

RESOLVED. The ten upstreamable commits of PR #390 boot and mostly work, but leave the multi-domain wedge in place. The missing piece is .236's #patchB5 (wiki Kernel Modifications Change 19, reclaim-and-retry on a domain-switch timeout). Ported minimally onto the clean series, orkd_dom_api goes FAIL 3/3 → PASS 3/3, with the recovery path observed firing. Patch: 24-rknpu-reclaim-leaked-domain-ref.patch.

The test bed

A second board, 10.3.0.235 (ROCK 5B+), was provisioned as a clean bed for the upstreamable series — see Board Provisioning — ROCK 5B+. Kernel built from the armbian rk-6.1-rkr5.1 tree at its last pure-armbian commit (1cd878b3ac49) plus exactly PR #390's ten commits, applied with git am --3way, no conflicts, zero warnings in rknpu/rockchip-iommu. Stock armbian config: no SRAM enabler, no FENCE, none of .236's ~49 diagnostic hunks. uname -r = 6.1.115-vendor-rk35xx-npu.

This matters because it is the first time the clean series has been booted and exercised. The PR's own notes say it had only been compiled against develop-6.1, with runtime testing done on the armbian tree carrying the full board state — which silently included the fix identified below.

ork-driver 1.0.165, make test under the guard, governors verified (DDR 2400 / A76 2400).

Arm 1 — PR #390 only

ORKD_DOM_API: FAIL        domain=1 / domain=2 : run/pack FAIL   (3/3 fail)
TEST_SLICE_RESCUE: FAIL   int4 rescue rc=-1

Kernel side, over netconsole — 6 occurrences:

RKNPU: switch iommu domain time out, failed to switch iommu domain, id: 1
RKNPU: rknpu_gem_object_create error
RKNPU: soft reset, num: 6

Everything else passed: TEST_BMM, test_offline_load, TEST_F16_LOAD, test_grouped_i8, SN3 GATE, TEST_SSD_CHUNK, TEST_SSD_CHUNK_NPU, TEST_MODE_TRANSITION, TEST_BMM_FUSED, STREAM INTERLEAVE, AFFINITY, TEST_F16COLSPLIT. The board survived — a correctness gap, not a stability one.

The mechanism, stated by PR #390's own code

rknpu_iommu_domain_get_and_switch() only switches when iommu_domain_refcount reads exactly zero, otherwise it spins to RKNPU_SWITCH_DOMAIN_WAIT_TIME_MS and returns -EINVAL. The comment the PR adds to rknpu_iommu_domain_put() describes the consequence precisely:

a bare atomic_dec() turns a single unbalanced put into a permanent wedge: the count never reads zero again, every domain switch burns its full RKNPU_SWITCH_DOMAIN_WAIT_TIME_MS and fails, and because rknpu_gem_object_create() switches domains, every subsequent allocation then fails with -EINVAL until the machine is rebooted.

PR #390 clamps the counter at zero so it cannot go negative — but clamping does not recover a count that is already too high. Once something leaks a reference, the device is wedged for the rest of the boot. That is exactly the observed time out + rknpu_gem_object_create error pair.

Arm 2 — PR #390 + reclaim-and-retry

Minimal port of #patchB5, without .236's debug counters (98-line diff, two files):

  • rknpu_job.c: rknpu_reap_all_cores() — rknpu_core_mask() and rknpu_job_timeout_clean() are static there, so the reaper must live in that file.
  • rknpu_iommu.c: a dom_reclaim module param (default 1) and, on timeout, reap the cores; if nothing is in flight but the refcount is non-zero, treat it as leaked, force it to zero and retry the switch once.
ORKD_DOM_API: PASS        3/3 bit-exact      <- was FAIL 3/3

and the recovery is observed, not inferred — one timeout this boot (vs 6), one reclaim:

RKNPU fdab0000.npu: RKNPU: reclaim: no jobs in flight but refcount=2 -- forcing to 0

So the leak still happens; it is simply no longer fatal. Verified in the condition that failed (the full suite), not in isolation.

Two honest limits

1. #patchB5 treats the symptom. refcount=2 with no jobs in flight means some path still fails to put, and that root cause is not found. The reclaim converts a permanent wedge into a bounded, recoverable anomaly — which is worth having, and is why .236 has it — but the leak is the real bug and is still open.

CLOSED the same day. The leaking path is rknpu_job_timeout_clean() freeing the jobs still QUEUED on a core's todo_list without releasing their dom_held reference; the fix is one test_and_clear_bit + rknpu_iommu_domain_put() in rknpu_job_free(). Measured: 3 leaks == the reclaimed refcount=3, and with the fix the whole suite logs zero domain-switch timeouts even at dom_reclaim=0. See Exp-2026-10-09 — the IOMMU domain-reference leak, root-caused. Branch rknpu-domain-ref-leak in oRKLLM/rk3588-kernel, off rknpu-iommu-rework.

2. A single pass in isolation proves nothing here, and nearly misled this investigation. Run alone on a fresh boot, orkd_dom_api passed on the unpatched PR #390 kernel too, because the switch never timed out — the reproducer is one-trial-per-boot and the wedge is created by the preceding tests leaking a reference. The causal claim rests on the full-suite comparison plus the reclaim: line, not on the isolated run.

Not caused by this change

  • TEST_SLICE_RESCUE (int4 rescue rc=-1) fails on both arms. It is the int4 doorbell path and is untouched by the reclaim. Confirmed 2026-10-09 on the STOCK armbian kernel (6.1.115-vendor-rk35xx, zero ork patches) with shape-for-shape identical results, so it is not PR #390 and not any of our kernel work — it is a .235 board property. See Exp-2026-10-09 — the IOMMU domain-reference leak and the run_i4_bchain_db row in OPS_REGISTRY.md.
  • TEST_F16COLSPLIT failed once in-suite on arm 2 at its largest shape (M=128 K=3584 N=18944) and passed on arm 1. It then passed 3/3 in isolation on the same kernel. Intermittent under suite load, pre-existing; not a regression from the reclaim. Do not read a single in-suite failure of this test as a regression without an isolated re-run.
  • Separately: TEST_F16COLSPLIT did improve with the library, not the kernel — it failed three shapes at ork-driver 1.0.164 and passes them at 1.0.165, so the fp16 K-slice fix also resolves fp16 doorbell misses.

A hypothesis this disproved

It was proposed earlier the same day that the failures were #patch73 (an async job must own its own power reference — a NONBLOCK submit whose completion IRQ is dropped because the NPU powered down under it). Wrong. The captured signature is the IOMMU domain switch timing out and rknpu_gem_object_create failing; #patch73's signature is a dropped completion on a powered-down NPU. Recorded because the hypothesis was stated before the capture existed.

Recommendation for PR #390

Add the reclaim-and-retry. Without it the series ships a known-reachable permanent wedge whose failure mode its own comments already describe. The 98-line patch is attached; it is minimal (no debug counters) and gated by a module parameter defaulting to on.

Method notes

  • .235 is now a usable bisect bed: panic=10 so a panic warm-reboots and the board returns unattended, panic_on_oops=1, netconsole to 10.3.0.239 so kernel output survives a crash without writing to the micro SD card.
  • Narrow a kernel bisect by reading, not by building. 53 patch markers at ~20 min per build/install/reboot/test cycle is impractical; diffing the single function that emits the failing message found the candidate in one step.
  • Raising the console loglevel to 7 (needed for netconsole to mirror ordinary messages) floods the HDMI console, most visibly with RKNPU: Get total_rw_amount is not supported on this device! — a LOG_WARN in rknpu_job.c that fires when config->amount_top == NULL and then returns 0. Benign, and it reads like a crash. Unbind the framebuffer console (echo 0 > /sys/class/vtconsole/vtcon1/bind) rather than lowering the loglevel, so netconsole keeps the diagnostics.