Follow-up to Exp-2026-10-09 — PR #390 alone does not fix the domain switch, which closed with two open items. The first — "#patchB5 treats the symptom. refcount=2 with no jobs in flight means some path still fails to put, and that root cause is not found" — is now closed.

rknpu_job_schedule() takes exactly one iommu_domain_refcount reference per job and records it in job->dom_held. PR #390 added three release sites, and they are complete for the jobs that reach them:

A fourth path frees jobs and releases nothing. After retiring the running job, rknpu_job_timeout_clean() drains the rest of that core's todo_list:

do {
        schedule_work(&job->cleanup_work);
        spin_lock_irqsave(&rknpu_dev->irq_lock, flags);
        if (!list_empty(&subcore_data->todo_list)) {
                job = list_first_entry(&subcore_data->todo_list, struct rknpu_job, head[i]);
                list_del_init(&job->head[i]);
        } else {
                job = NULL;
        }
        spin_unlock_irqrestore(&rknpu_dev->irq_lock, flags);
} while (job);

Every job on that list was put there by rknpu_job_schedule(), so every one holds a reference — and rknpu_job_cleanup() → rknpu_job_free() never looked at dom_held. One leak per queued job reaped.

An audit of all eight rknpu_iommu_domain_get_and_switch() call sites found this to be the only unbalanced one. The other seven (rknpu_gem_object_create, rknpu_gem_object_destroy, rknpu_gem_destroy_ioctl, rknpu_gem_sync_ioctl, RKNPU_ACT_SET_DOMAIN, RKNPU_SET_IOMMU_DOMAIN_ID, rknpu_iommu_free_domains) put on every exit including the error paths, and a failed get takes no reference, so the return in the three-strike retry loops is correct.

ork-driver's own ork_npu_reap_stuck() (src/npu/core/device.c), which exists precisely to drive rknpu_job_timeout_clean. It submits a NONBLOCK fp16 dummy per core (flags=0x1|0x2, timeout=300) and logs ork: reap-stuck: fp16 nonblock dummy core=N (trigger timeout_clean).

rknpu_submit() calls rknpu_job_timeout_clean() before scheduling, but it only reaps a job that is already past its own timeout. When the stuck core's job is not yet past it, the dummy does not run — it queues. The BLOCKING drain that ork_npu_reap_stuck issues next does not retire it either: that job is itself aborted, and rknpu_job_abort() unlinks only itself. So the dummy sits in todo_list until some later timeout_clean, which frees it — and takes its domain reference away with it. Three cores, so up to three per episode; the measured counts were 2 and 4.

This is why the reproducer is the full make test and not orkd_dom_api: the leak is created by the earlier tests that drive recovery (i4, test_slice_rescue, the fp16 colsplit drop path), and orkd_dom_api is merely the first thing afterwards that needs a domain switch.

A per-call-site counter array (rknpu.dom_dbg, module param, built-in → visible at /sys/module/rknpu/parameters/dom_dbg) was added to the 13-commit series, together with the candidate fix behind rknpu.dom_fix so one kernel serves both arms. Indices: [0] schedule gets, [1] done puts, [2] abort puts, [3] reap-of-running puts, [4] jobs freed while still holding a reference, [5] queued jobs reaped, [6] frees.

dom_dbg = 24224,24192,0,42,3,3,24237,0
[0] schedule gets          24224
[1] done puts              24192
[2] abort puts                 0
[3] reap-of-RUNNING puts      42
[4] freed STILL HOLDING a ref  3     <- the leak
[5] QUEUED jobs reaped         3
[6] rknpu_job_free calls   24237

same run, kernel log:

RKNPU: switch iommu domain time out, failed to switch iommu domain, id: 1
RKNPU: reclaim: no jobs in flight but refcount=3 -- forcing to 0

[4] == [5] == the reclaimed refcount == 3. Every job the reap freed off a todo_list arrived at rknpu_job_free() still holding its reference, and the count of them is exactly the residue the reclaim had to force to zero. (An earlier run on the same kernel reclaimed refcount=4; the count tracks how many dummies happened to be queued, which is why it moves between runs.)

dom_dbg = 24066,24046,0,49,3,3,24098,0
[4] freed STILL HOLDING a ref  3     <- the same three reaps happened ...
[5] QUEUED jobs reaped         3     <- ... and every one was released
ORKD_DOM_API: PASS

The whole run's kernel log (912 lines) contains zero switch iommu domain time out, zero reclaim: ... forcing to 0, zero failed to switch. The fix alone closes it; the reclaim never has to fire.

Board 10.3.0.235 (ROCK 5B+), 6.1.115-vendor-rk35xx-npu = armbian 1cd878b3ac49 + all 13 PR #390 commits, DDR 2400 / A76 2400, ork-driver 1.0.165, full make test under the NPU guard.

Unchanged and identical across all three arms, so caused by none of this: TEST_SLICE_RESCUE, TEST_F16COLSPLIT, the four i4 shapes, test_speed's 3637 vs 3600 us limit, and 4x UNEXPECTED ACT_SET_DOMAIN(import) failure errno=22 in test_f16_load.

Release it in rknpu_job_free() — the single destructor every teardown reaches, and the same place and shape as the fence rescue PR #390 already put there (aedcff95):

	if (test_and_clear_bit(0, &job->dom_held))
		rknpu_iommu_domain_put(job->rknpu_dev);

test_and_clear_bit() keeps it at most once per job, so the three existing release sites are unchanged and the normal completion path is unaffected: by the time cleanup_work runs on a job that completed, rknpu_job_done() has already cleared the bit.

Prepared as a single commit on oRKLLM/rk3588-kernel branch rknpu-domain-ref-leak, off rknpu-iommu-rework (PR #390). Not fast-forwarded into the PR branch.

#patchB5's reclaim-and-retry is still worth shipping with this. It is the backstop for any future unbalanced path, and it is what turns such a path from a permanent wedge into a logged anomaly. With the leak fixed it should simply never fire — which is itself the regression test.

After the measurement above, .235 was rebuilt per a direction change onto the SAME source state .236 runs (board-snapshot-2026-10-09, 8fac2134ab53) + .235's stock armbian .config + CONFIG_ROCKCHIP_RKNPU_SRAM=y + the leak fix above. Boot verification passed:

The A76 OPP ungating is NOT in the current board snapshot. Booted without user_overlays=rk3588-a76-2400, .235 came up at cpu4=2304 / cpu6=2352 — PVTM-binned, i.e. still gated. Decompiling the installed rk3588-rock-5b-plus.dtb shows opp-2304000000 at <0xf9 0x24> and opp-2400000000 at <0xf9 0x80>, and the tree's own rk3588s.dtsi cluster1-opp-table ungates only up to opp-2208000000, leaving 2256/2304/2352/2400 gated. So the statement on Board Provisioning — ROCK 5B+ that .236's DT carries <0xf9 0xffff> on all four does not hold for this snapshot. The overlay was restored and remains the mechanism that gets .235 to 2400.

Then the first NPU workload on that kernel hard-wedged the board. ./i4 under the guard, with rknpu.dom_reclaim=0, went: allocate iova then soft reset, num: 6, three HEALTHY commit snapshot / HEALTHY PCBLK diagnostic dumps, soft reset, soft reset, then silence at 99.9 s uptime. No panic, no oops, no further netconsole output, no ping, no ssh. panic=10 cannot help because nothing panicked.

Two things make this worth recording rather than filing as noise:

• The same ./i4 shapes merely fail gracefully (rc=-1, recovery exhausted) on both the 13-commit PR #390 kernel and the stock armbian kernel on this board. On the board-tree kernel the first such failure took the machine down inside 100 s.
• There is no remote power control for .235. The only available Home Assistant outlet is Rock 5B Plug, which is .236. The Office outlet Radxa Rock 5B Blue - 32GB reads unavailable. Recovery needs a physical power cycle — and per the provisioning page, a hard cut during an NPU submit is exactly what can corrupt the SPI bootloader.

The board was reset by hand and came back on the same kernel (uname -r 6.1.115-vendor-rk35xx-npu, DDR 2400, A76 2400/2400, SRAM region present). The kernel boots; the wedge was a runtime NPU/IOMMU lockup ~100 s into a workload, not a boot failure.

make test's full list minus i4 and test_slice_rescue (the two that drove the wedge), same order, under the guard: fail=0 — everything passes, including ORKD_DOM_API, orkd_seq_probe, chain_xition_probe, test_bmm_fused, chainrr_conc_probe, test_ssd_chunk_npu and test_mode_transition. Kernel log for the whole run: 0 reclaim, 0 domain-switch timeouts, 0 failed to switch (53 soft resets, the ordinary per-call ACT_RESETs). So the leak fix holds on this tree as well as on the PR #390 one.

Two things got better relative to the 13-commit PR #390 build on the same board:

Not attributed to SRAM without an A/B — the two kernels differ by the whole board tree, not just the SRAM config line.

Still unrun on this kernel: i4 and test_slice_rescue. They are the known wedger here, and .235 has no remote power control, so re-running them costs a human walking over to press RST if it goes the same way. Decide that deliberately rather than by habit.

With the board back, i4 and test_slice_rescue were run on the board-tree + SRAM kernel.

SRAM changes nothing. freeSRAM=956KiB now appears in the dumps (was 0KiB) and the same four i4 shapes fail with the same signature. So the SRAM enabler is not the variable between .235 and .236.

Two independent causes, separated by probe.

1. The bank-width N-tile Wb = (131072/K) & ~63 is too wide on .235. M=8 K=512 at N=64 (Wb→64, Wmax=1) passes; at N=256 (Wb=256, Wmax=4) it fails; forcing ORK_I4_WB=64 on the N=256 shape makes it pass. Same for M=32 K=1024 N=128 (Wb=128). With ORK_I4_WB=64 alone:

Every test_slice_rescue golden matches the .236-generated value (0x42a675c1…, 0x0166d951…, 0xf1e54fbd…, …). Matching a golden computed on the other board is what proves the output is identical, not merely non-erroring — a plain rc=0 would not.

Wmax>1 is not universally broken: M=64 / 128 / 256 at K=512 N=256 pass at the default Wb=256. So the envelope depends on the M-group height as well as the tile width, and the exact law was not derived — do not guess one from these points.

2. H=8 at K=2048 is an overshoot on .235. i4_hcap_probe at K=2048 N=64 M=16: H=2/3/4 agree on one checksum; H=6, 8 (the table value), 16, 24, 32, 48, 64 all fail. That is what the two remaining i4 shapes hit; test_slice_rescue survives them through its resubmit loop (7 recoveries in the passing run).

A blanket ORK_I4_H override is UNSAFE, and this is the part to remember. The knob is global, so ORK_I4_H=4 also overrides K where the measured ceiling is lower. Under ORK_I4_WB=64 ORK_I4_H=4 every shape runs — all 16 i4 shapes report maxerr=0 — but test_slice_rescue's i4-wideK (K=10240) and i4-rem2560 (K=18944) return rc=0 with a wrong golden. An H fix has to be a per-K table edit, never the env knob. Same lesson one level down: at K=2048 N=64 M=16, H=12 returns rc=0 with a different checksum. Grade a candidate H by checksum, never by rc.

The earlier hard-down and a second, cleaner event during test_slice_rescue have the same cause, and the board tree's own diagnostics name it:

RKNPU: core 0: NOT dispatching -- still owned by job 000000002f62a97c (flags 0x2, age 2105810us,
       int_cnt 1, run_cnt 0); queued work will NOT run until it retires
   ... same for core 1 and core 2 ...
rockchip-pm-domain fd8d8000...: failed to set idle on domain 'npu1', target_idle = 1, val=0
Kernel panic - not syncing: panic_on_set_idle set ...
  rockchip_pmu_set_idle_request <- rockchip_pd_power_off <- genpd_runtime_suspend
  <- pm_runtime_put_sync <- rknpu_power_off <- rknpu_power_off_delay_work

The int4 overshoot strands one NONBLOCK job per core (flags 0x2, run_cnt 0 — the hardware never started them). They are never retired. ~93 s later the process exits, the power-off delay fires, and the driver powers the genpds down with cores still non-idle; npu1 will not idle and the stock vendor panic() in drivers/soc/rockchip/pm_domains.c kills the machine.

Two things to be precise about:

• The panic() is stock. git diff of pm_domains.c between the armbian base 1cd878b3ac49 and the board snapshot is empty; the base already has panic("panic_on_set_idle set ..."). It is not one of the ~49 diagnostic hunks.
• But only the board tree reaches it. failed to set idle on domain 'npu1' appears exactly once in the whole netconsole history, on the board-tree kernel — never on the PR #390 kernels or on stock, across many runs of the identical failing shapes. The board tree's power/reset rework is what lets power-off run with cores still owning jobs. The watchdog is not involved (wd_period_us=0, wd_abort_ms=0 by default) and neither is the per-core reset (it fires only on RKNPU_JOB_STALLED).

panic=10 turned the second event into an unattended warm reboot, which is why the board came back on its own; the first one did not, and needed an operator.

Practical consequence for .235: on the board tree, an int4 overshoot is not a failed test, it is a reboot. Setting ORK_I4_WB=64 removes two of the four shapes and makes test_slice_rescue pass; the K=2048 H overshoot still needs a per-K table fix before .235 can run int4 prefill on the parity kernel safely.

The obvious reading of "the constants do not transfer between two RK3588s" is that they are a property of the die. They are not. With the geometry ladder disabled (ORK_I4_NOLADDER=1), one known-bad shape (K=2048 N=64 M=16, i.e. the table's H=8) and one known-good control (K=512 N=64 M=16) were run while the NPU devfreq and then the DMC devfreq were capped:

The NPU clock is irrelevant across its whole range; the DRAM clock decides it. And f28a3df8f9118d1d is the correct answer — the same checksum H=2/3/4 produce and the same one the ladder arrives at — so the lower DDR clock is not merely "not failing", it is computing the right thing.

Control, because switching the DMC governor to userspace is itself a change: holding the governor at userspace and varying only the frequency gives 2400 → rc=-1, 1968 → rc=0, 2400 → rc=-1 again. Frequency is the variable.

• It is a DMA/timing envelope, not a CBUF capacity one. A capacity limit cannot care how fast DRAM runs, and would not be unmoved by a 2.5x change in NPU clock.

• The DIRECTION rules out starvation, and an earlier revision of this page had it backwards. FASTER memory is what breaks it: 2400 fails, 1968 and 1320 work. "The DMA cannot keep up" predicts the opposite and is wrong. More bandwidth and lower latency breaking a fixed-function pipeline is the signature of a race that the faster delivery wins — data or a descriptor arriving before the sequencer is ready to own that buffer — not of a starved one. The 0x1040 bank split the registry already describes is a bank-OWNERSHIP mechanism, and its documented failure modes ("poison pill", "computes row 0 only") are the shape a lost ownership handshake would take. That is a hypothesis, not a measurement; what IS measured is the direction, and the direction excludes starvation.

• It is the clock, not the DRAM type. .235 is LPDDR5 at 2400, 1968 AND 1320 — only the clock changes across those three, and only the 2400 case fails. So "LPDDR5 behaves differently from LPDDR4X" is not the explanation on its own.

• It is an interaction, not a blanket speed limit. The control shape (K=512 N=64 M=16) is correct at 2400. Only some tile geometries break, so the race window depends on how the program walks banks, which is why it presents as a per-(K, M, N) envelope rather than "this board is too fast".

• Why the table held until now. .236 is LPDDR4X at 2112 MHz, below the threshold, which sits between 1968 and 2400. .235 is LPDDR5 at 2400 MHz, above it. Every board this project had ran under the threshold, so a DRAM-frequency term in the envelope was invisible. This is a sampling limit, not a modelling error — with one memory configuration it could not have been seen.

• Wb is a different story and IS a modelling gap. Unlike the H table, Wb is documented as architectural — one CBUF weight bank, "DO NOT RAISE IT" — and a pure bank-containment rule has no M term. But Wb=256 fails at M=8 and works at M=64 at the same K and N. So the real constraint couples tile width to M-group height, and that coupling is in neither board's model.

orki_i4_hcap() is valid for DDR at or below ~2112 MHz. It is not a SoC constant and must not be treated as one. Re-measuring it per DRAM clock is possible but would need a table per memory configuration; the geometry ladder below handles it without one, by discovering at run time that a geometry will not take and stepping down.

Because the envelope has a DRAM-frequency term that no table captures, the durable fix is not a second table — it is to discover at run time that a geometry will not take, and step down. The entry point now runs the batch up to three times:

1. the measured geometry (today's behaviour, and the only step a board under the threshold ever runs),
2. the narrowest N-tile (Wb=64),
3. that plus a shorter M-batch (H=4).

Both steps are bit-exact — same tiles, same kernel, only the grouping changes — and a step that would not change the geometry is skipped, so a board where the measured constants hold pays nothing. ORK_I4_NOLADDER=1 disables it for an A/B.

H is only ever LOWERED. Raising it past orki_i4_hcap() does not fail, it miscomputes silently, which is the trap a blanket ORK_I4_H override walks into (it makes all 16 i4 shapes report maxerr=0 while i4-wideK and i4-rem2560 return rc=0 with the wrong golden).

Every retry is preceded by ork_npu_mode_reset(), and that is not optional. The first cut of the ladder had no reset and turned test_slice_rescue's i4-refuse from a miss into a wrong answer — 308202 mismatches between the natural and forced arms. A geometry the part will not take leaves CBUF residue and the next attempt computes against it. This is the same stale-CBUF-residue mode the orki_i4_hcap K=2560 entry already documents. With the reset in place the case reports OK (natural refuse->rescue, rc=0, ==forced, golden 0x928fbeeca9293205).

Result on .235, with no env knobs at all:

The goldens matching is the proof: they were generated on the other board, so the ladder is arriving at the same numbers, not merely at a number.