Exp 2026 07 16 Dynamic Submit API and Doorbell - oRKLLM/ork-driver GitHub Wiki

Exp 2026-07-16 β€” Dynamic-Submit API, Doorbell Progress, and the task_number Ceiling

A host-driven, NONBLOCK submission layer over the rknpu chain path: submit a matmul chain and watch it, steer it, and free it mid-flight from userspace on the stock kernel (no MMIO, no driver change). This page documents the submission model it exposed, the API (ork_dyn_* / ork_dyn_queue_* / ork_pc_* in src/npu.c), the two validated wins, and the negative results that pin down exactly what the hardware/kernel will and won't allow. Board 10.3.0.236, RK3588, kernel 6.1.115-vendor-rk35xx-sram.

1. The submission model β€” a fixed-length track

A SUBMIT ioctl hands the kernel a task list (array of rknpu_task, each with a regcmd_addr + regfg_amount), the regcmd buffer (the programs; each ends with a descriptor at a fixed word: 0x0010 = next-program address, 0x0014 = next-program size, 0 = "I am terminal"), and task_number.

The load-bearing fact, established by measurement:

A submitted job runs exactly task_number programs and then completes. task_number is fixed at submit time and is immutable. The kernel's completion bookkeeping is locked to it.

This yields a hard asymmetry:

  • You can run FEWER programs than declared β†’ HALT works: writing 0x0014 = 0 into a future program's live regcmd (doorbell-timed, before the sequencer reaches it) makes it terminal β†’ the chain stops there and frees the NPU early. Validated: halt(8) on a 16-op chain β†’ outputs 0..8 computed, 9..15 never run.
  • You can NEVER run MORE programs than declared β†’ LOOP and EXTEND both fail (see Β§5).

The forward advance between programs is task-indexed and bounded by task_number β€” not a runtime jump that follows the in-regcmd 0x0010 pointer wherever it points. Decisive measurement: a chain submitted with task_number = 1 runs only program 0 even though program 0's 0x0010 points at program 1. If the hardware blindly followed 0x0010, it would have continued; it did not.

2. The doorbell β€” host-observable progress, NONBLOCK

Submit with flags |= RKNPU_JOB_NONBLOCK (0x2) so the ioctl returns before the HW finishes. Each program's output slot doubles as a progress doorbell: seed every output's last element to a sentinel (0x7fffffff) before submit, then busy-poll it with dc civac (invalidate β†’ read from DRAM). When a slot holds a real value the op that wrote it has completed. ork_dyn_progress = highest completed op index. Overhead is a few Β΅s/op; per-op progress is cleanly observable in submit order (validated by tools/doorbell_id_probe, tools/static_id_probe).

3. The API (src/npu.c, header include/ork_npu.h)

  • ork_dyn_begin(c, S, tasks) β€” build S int8 matmul programs, submit NONBLOCK (single-core chain).
  • ork_dyn_begin_mc(c, S, tasks, nc) β€” multi-core: partition across nc cores, one NONBLOCK submit per core (Β§4).
  • ork_dyn_progress(h) / ork_dyn_halt(h, at) / ork_dyn_end(h) β€” poll / early-exit / drain+writeback.
  • budget: ork_dyn_max_steps() / ork_dyn_steps(h) / ork_dyn_remaining(h).
  • ork_dyn_queue_* β€” a chunk-pipeline: accumulate tasks, run a chunk NONBLOCK while the caller does other work (CPUβ€–NPU decode split), auto-split work > chunk_max into successive clean chunks. ncore selects single/multi-core; linger defaults to one submit floor (~167 Β΅s β€” coalescing past a floor isn't floor-bound, so lingering longer buys nothing).
  • ork_pc_compile/run/free β€” precompiled-program cache (Β§6).

v1 constraints (all paths): M = 1, DT_I8, K % 512 == 0 && K <= 4096, output C resident in an ork_dma_alloc buffer (for the doorbell). Input A is copied into a scratch DMA buffer β€” zero-copy A miscomputes at M=1 (the known ZC-A bug), so it is never read in place.

4. Multi-core NONBLOCK stream β€” the DISPATCH-half win

ork_dyn_begin_mc partitions the S matmuls across the NPU cores, builds each core's sub-chain into its per-core buffers (c->mrc[i]/mtk[i]/maf[i], the same buffers mcworker/run_stream_i8 use), and submits each NONBLOCK on core_mask = 1<<i. Core-partitioned concurrency is the proven-safe pattern (run_stream_i8 already runs the 3 cores concurrently via blocking pthreads); NONBLOCK simply lets the host poll doorbells instead of parking threads. No wedge observed.

Board head-to-head (N=16–32, M=1, int8, NPU-only wall time):

path time vs single vs blocking stream
single-core queue (begin) 988 Β΅s 1.0Γ— β€”
multi-core NONBLOCK (begin_mc) 436 Β΅s 2.27Γ— 1.38–2.30Γ—
blocking 3-core stream (ork_mm_run_stream_i8, the #14 dispatch) 602 Β΅s β€” 1.0Γ—

Multi-core NONBLOCK beats even the blocking 3-core stream (1.38–2.30Γ—) because the doorbell rendezvous has no thread-join round-trip. Commit b9ce004; bench tools/ork_dyn_queue_bench.c.

5. Negative results β€” LOOP and EXTEND are not viable on the stock kernel

Both attempts to run more than task_number programs failed, consistent with Β§1:

  • EXTEND (ork_dyn_append, redirect 0x0010 forward into a program beyond the submitted count): the HW walks into territory the kernel's accounting doesn't know about β†’ RKNPU: job abort, ret: -22, ~10 s timeout, soft-reset. Kept behind ORK_DYN_RESERVE + opt-in flag as documented negative-result RE.
  • LOOP / persistent spin (ork_dyn_spin_probe, circular chain: program 0's 0x0010 self-loop to keep a job alive for zero-resubmit feeding): the sequencer walked forward instead of looping (task-indexed, bounded by task_number), and the backward 0x0010 disagreed with the forward task list β†’ corrupt output (0/8 correct), soft-reset. tools/dyn_spin_probe.c.

Conclusion: 0x0010 redirect is not a runtime jump you can steer; only 0x0014 = 0 halt is a working dynamic control. A true persistent/looping job would need a driver re-arm/loop change or MMIO PC control β€” not reachable from userspace on the stock kernel. steer_probe only ever validated halt; generalizing it to "redirect works" was the error these two probes corrected.

The wrap is therefore chunk + resubmit β€” each chunk a clean task_number-bounded job; the inter-chunk bubble is ~one submit floor (167 Β΅s), < 0.1 % at large chunks. Multi-core NONBLOCK makes each chunk's dispatch cheap enough that a persistent job isn't needed.

6. Precompiled-program cache (regime A) β€” the BUILD-half win

The per-token host build β€” synth_i8 + validate_regcmd + memcpy of ~108 regs per program + per-op scratch-A alloc/free churn β€” is the reducible ~95 Β΅s of the submit floor. For a fixed chain with pinned buffers the program is identical every token (weight/shapes/addresses baked in); only the activation contents change. So ork_pc_compile builds the S-program chain once into a static pool (the task list references the pool β€” a "program domain" indirection, like a page table over scattered pages), and ork_pc_run re-runs it with just an A-refresh + submit.

Board bench (S=16, M=1, int8, single-core), tools/ork_pc_bench.c:

per token
synth-every-call (ork_dyn_begin) 2652 Β΅s
precompiled (ork_pc_run) 440 Β΅s β†’ 6.03Γ—
compile (one-time) 329 Β΅s

T_pc β‰ˆ the raw single-core NPU floor β€” the host build is essentially eliminated. Commit 83ea8b2.

Two regimes: A = fixed dense decode chain (built, above); B = varying MoE (a precompiled per-expert program pool, composed per token by selecting descriptors β€” the literal "compose a domain from cached pages" case; not yet built).

7. How the two halves compose

Multi-core NONBLOCK cuts the dispatch half of the floor; the precompiled cache cuts the build half. They stack. The decode application is the swap-hidden #14 split: the NPU runs its int8 sensitive share via a multi-core NONBLOCK (precompiled) chain while the CPU runs the int4/NF4 bulk in parallel, rendezvousing on the doorbell β€” the NPU share hidden behind the CPU window at near-zero added time.

7b. M>1 and fp16 doorbell β€” what the completion word actually tracks

Generalizing the doorbell from M=1 to M>1 and from int8 to fp16 exposed that "poll the last output element" is precision- and layout-specific. Findings (board 10.3.0.236, tools/ork_dyn_test.c D=int8-M>1, E=fp16-M>1):

  • int8 M>1 β€” poll every row's last col. A single global-last-element (M*N-1) doorbell RACES: the M-tile scheduler does not write row M-1 col N-1 strictly last across rows. Within a row the last col is written last, so ork_dyn_done_i polls all M rows' last cols (M≀64, cheap). Validated D 16/16 across ~80 runs (single-core A M=1 likewise). Commit 62e937c.

  • The fp16 "output write-order permutation" hypothesis is DISPROVEN. A seed-ALL-elements + snapshot probe (seed every M*N output word to the sentinel, fire the round, and the instant the row-last-col doorbell reports done, snapshot which words are still sentinel) found zero elements written after their row's last col. fp16 does not tile the output surface differently from int8; the doorbell word is correct. (Cross-checked against the docs: NPU-Quirks records fp16 mode/warm and 0x1040 schedule miscompute behavior, and the "C[0] correct / C[last] wrecked" coherency trap β€” but no fp16 output write-order quirk. rocket_registers.h/NVDLA give the DPU output-stage 0x40xx strides identically for both.)

  • The real fp16 failures are coherency + cold-mode, not ordering. Two effects, both fp16-only:

    1. Clean-before-write. fp16 direct-output needs every output cache line cleaned to DRAM before the NPU write (dc cvac), or dirty/stale host lines in the output interior race the NPU's writes and corrupt it β€” the same DMA cache-coherency class as the ORK_ZC_OUT fix. ORK_MC_SEED's fp16 branch now seeds every element (not just last-cols). This alone moved E from computing wrong ~80 % of runs (out[0][0]=0x0, "done" in ~14 Β΅s) to out[0][0]=0x44000000=512.0 correct.
    2. In-submit drain. fp16 begin_mc polls the real round to completion before returning (int8 stays truly async β€” the doorbell win is int8's); deferring the first poll to ork_dyn_end let it read the throwaway warm round's stale last-cols.

  • Residual, unresolved: a systemic ~10–20 % "all-16-fail" race β€” and it is NOT the warmβ†’real handoff. When it fails, most/every task is wrong (0/16, 1/16, 6/16 all seen). Initially read as a warmβ†’real double-submit collision on the shared per-core buffers, but every warm-handoff lever was tested and none fixes it: re-ork_npu_enter between the warm and real rounds (16/20, unchanged), full-surface warm-drain before reseed (regressed to 19/25), extra post-round settle (18/20), and β€” decisively β€” a blocking (NONBLOCK-flag-dropped) fp16 warm submit that guarantees the warm job retires before the real round: it ran clean (no EINVAL/soft-reset on the mc path, task_number=P per core) but regressed reliability to ~53 % (8/15). Two further levers, prompted by "we already have chains that swap int8↔fp16 reliably (run_multicore/run_stream_f16) β€” what do they do differently?", were then ported into begin_mc and both plateau at ~85 %, no better than the NONBLOCK doorbell:

    • Blocking establishment = run_multicore's exact protocol β€” reps = mwarm?1:2 BLOCKING passes per core (flags w/o 0x2), bsync FROM_DEVICE after each, same HW-chain + ork_ppflags()=0x5 ping-pong: 21/25 (84 %). (Confirms ping-pong and HW-chaining are NOT the differentiators β€” run_multicore uses both and is reliable.)
    • SW-chain handoff β€” submit each task standalone (task_number=1, subcore_task={p,1}, no in-regcmd 0x0010/0x0014 walk) instead of HW-chaining P: also flaky (12/16 seen).
    • Neutral all-zero fp16 "MAC-array reset" primer(s) between warm and real (a K=512,N=16 zero matmul submitted nΓ— per core, blocking): 1Γ—=flaky (10/16), 2Γ—=18/20, 4Γ—=13/20. Non-monotonic and inside the noise band β€” the 2Γ— "90 %" was luck, not signal; priming the MAC array does not control the failure.
    • SDP-stage primer (a neutral all-zero ork_npu_ewmul_f16 + re-establish fp16 matmul mode + cold re-warm, before the real round): 1/4 β€” worse. Priming the output-processor stage does not help either.
    • Per-core scratch + copy-back structure (force direct=0 so each core writes its own mcc[i] scratch, single-threaded copy-back in end() β€” i.e. run_multicore's output structure instead of the shared in-place buffer): 4/20 (20 %) β€” dramatically WORSE. So the shared-in-place output buffer is not the culprit; the direct in-place path is in fact the best fp16 config found (~85 %).

    So the residual fp16 flakiness is not the warm handoff, not blocking-vs-NONBLOCK, not the HW-chain-vs-SW-chain handoff, not MAC-array carryover (matmul primers), not SDP-stage carryover (ewmul primer), and not the shared in-place output buffer (per-core scratch is worse). Across ~10 distinct establishment/handoff/priming/structure hypotheses the direct-in-place NONBLOCK path (~85 %) is the best; nothing reaches bit-exact reliability. The failure is timing/state-dependent in a way no host-side ordering, priming, or output-placement change reaches β€” the reliability that run_multicore has comes from its whole concurrent-pthreads execution model, not any transplantable knob. Verdict: fp16 multi-core is a job for the proven sync run_multicore/run_stream_f16; the async doorbell stays an int8 lever and fp16 begin_mc stays gated. (Definitive resting point, ~10 hypotheses exhausted.) What run_multicore has that begin_mc does not is the whole execution structure (per-core pthreads running concurrently, per-core scratch CC + a scatter-copy to the caller's buffer, and N-slice chains), not any single submit-flag/handoff knob. fp16 computes bit-exact when the real round is polled to completion in-place (probe: 16/16); the divergence is in the fast in-begin_mc path. So fp16 begin_mc stays gated behind ORK_DYN_F16; the default async path is int8-only and correct, and callers needing fp16 multi-core should use the proven sync run_multicore/run_stream_f16 (the async doorbell is an int8 decode lever). Open lever: give begin_mc fp16 its own per-core scratch + threaded structure (i.e. become run_multicore with a doorbell) rather than reuse the int8 in-place chain.

  • mc ork_dyn_end fix (general). The 500 Β΅s-no-progress early break in end() exists only to detect a single-core halt/append that stopped a chain short β€” both reject h->mc. For an mc chain a plateau just means cores haven't caught up, so the break bailed with n<S; it is now gated !h->mc (mc waits for all-done or the 3 s hard timeout). Fixes a latent early-exit for mc chains (int8 and fp16).

The failure is a K-reduction SHORT-COUNT (not coherency) β€” the actual mechanism

Classifying every output element on a failing run (tools/ork_dyn_test.c E, A=B=1 so each dot must = 512.0) pins the nature down precisely β€” no element is ever sentinel(unwritten), zero, or random garbage; every wrong value is 512.0 minus an exact multiple of 32:

mode signature arithmetic
systemic 0/16 ALL 8192 elems = 320.0 (0x43a00000) uniformly 512 βˆ’ 192 = 512 βˆ’ 6Γ—32
near-miss 15/16 exactly one M-row (256 elems) = 480.0 (0x43f00000) 512 βˆ’ 32 = 512 βˆ’ 1Γ—32

fp16 reduces K in K/32 = 16 passes (0x201:0x1044 = K/32); a value of 512 βˆ’ nΒ·32 is the reduction dropping n of the 16 K-sub-blocks. So the fp16 mc failure is an arithmetic under-accumulation in the K-reduction schedule (0x201:0x1040/0x1010 β€” the fp16 M-scheduler / CBUF-residency regs NPU-Quirks already flags as fp16's load-bearing/fragile ones), not a doorbell, cache-coherency, or mode-establishment fault. Systemic = the whole round under-counts K uniformly (a global-schedule miss); near-miss = a single M-row's reduction drops one block. This is why none of the ~10 establishment/handoff/ priming/structure levers helped β€” they can't fix a reduction that drops K-blocks. Next lever (grounded, untested): the fp16 concurrent path's CBUF residency / 0x1040 K-schedule β€” e.g. force full-K residency (0x1010) or sweep 0x1040 via ork_f16_fuzz_add(0x201,0x1040,Β·) for the M=2 concurrent case; the win that made run_multicore fp16 reliable may simply be that its per-core layout keeps the full K resident.

Desk register comparison vs the reliable path β€” the K-schedule regs are IDENTICAL

Following the short-count classification, the K-reduction registers were compared desk-first (logged from synth() via ORK_SYNTH_LOG) between begin_mc's fp16 path and the reliable make-test fp16 path, for the E-test shape (K=512, N=256, M=2):

  • begin_mc fp16: synth(mc=2, K=512, sched=1, cbuf=32768) β†’ 0x1040=162 (0xa2), 0x1010=48, 0x1044(Kpass)=16.
  • synth() is a shared function; run_multicore calls it with the same mc/K/sched, and 0x1040 depends only on K (mg=1 for mc≀64), 0x1010 on min(mc+1,R) with R=pow2⌊cbuf/KβŒ‹β‰₯3. So for this shape the two paths produce byte-identical K-schedule registers. "Match the reliable config" does not apply β€” the config already matches.
  • But the reliable make-test fp16 shapes never exercise K=512/sched=1 M>1. test_matmul's fp16 runs K=1536/2048 β†’ sched=0 (K=1536 non-pow2; Kβ‰₯2048 out of the [128,2048) window). So the fragile sched=1 0x1040 M>1 fp16 path is not validated-reliable anywhere; begin_mc is using an unproven fragile path, not diverging from a proven one.

Two register-grounded board experiments then ruled out the schedule value and concurrency (paced, no wedge): forcing sched=0 for begin_mc fp16 (0x1010=48, no 0x1040=162 override) β†’ still flaky (4/16, 0/16, 10/16); forcing single-core (nc=1, no 3-core DMA contention) β†’ also flaky (13/20). So the short-count is not the 0x1040 value and not cross-core concurrency. Since the systemic mode is uniform (all 16 tasks drop the same last 6 K-blocks β†’ 320) and it persists single-core, the residual is a runtime weight/activation-DMA-vs-reduction race intrinsic to begin_mc's chained fp16 setup β€” the K-blocks the MAC consumes aren't all delivered, non-deterministically β€” which the standalone probe_f16_mm (task_number=1, ACT_RESET-per-call, 2 blocking reps) avoids but no begin_mc-shaped variant reproduced. probe_f16_mm stages the activation identically (raw [M][K]), so it is not an activation-layout bug. Net: registers are not the lever; fp16 stays gated. Deeper root-cause = why the chained fp16 reduction intermittently under-consumes K, a runtime-DMA question left open.

Board incident

One hard wedge (no ping/ssh) during a rapid-fire loop of the flaky fp16 test β€” the early-exit freed scratch buffers while the mis-executing fp16 job was still in flight (IOMMU fault). Recovered by an HA power-cycle of the Rock 5B Plug (no SPI corruption; back in ~90 s). Once fp16 drains to completion before free, back-to-back runs are clean.

8. Board-safety note

The 0x0010-redirect probes (append, spin) each left the NPU/IOMMU fragile via repeated RKNPU: soft reset events; a later op then hard-wedged the board (no ping/ssh), recovered by an HA power-cycle of the smart plug (no SPI corruption). The wedge source is specifically the redirect experiments; the validated paths (begin_mc, ork_pc, halt) run clean. See also NPU Hard-Wedge Circumstances.

Code

  • src/npu.c: ork_dyn_begin / ork_dyn_begin_mc / ork_dyn_progress / ork_dyn_halt / ork_dyn_end / ork_dyn_append (experimental) / ork_dyn_spin_probe (negative) / ork_dyn_queue_* / ork_pc_*.
  • Tools: ork_dyn_test.c, ork_dyn_queue_bench.c, ork_pc_bench.c, dyn_spin_probe.c, steer_probe.c, doorbell_id_probe.c, static_id_probe.c.
  • Commits (feat/tiered-i4-residence): b94a4aa (API + queue), b9ce004 (multi-core NONBLOCK), 83ea8b2 (precompiled cache + spin negative result).