Exp 2026 07 14 Chained 2 Input SDP ERDMA - oRKLLM/ork-driver GitHub Wiki

Exp 2026-07-14 — Chained 2-input SDP (ERDMA) & the single-submit attention chain

Goal: collapse the on-NPU attention (QKᵀ → max → exp → Σ → normalize → A·V, all already coherent as separate submits) into ONE task_number>1 PC-chain (M4). Blocked on getting a 2-input SDP (a chained op that reads a second operand via ERDMA — the per-channel normalize and the EW-max both need this) to run as a non-first chained task. Board RK3588 10.3.0.236; ork-driver feat/attn-primitives.

What works / doesn't

  • 1-input SDP chains fine. mm→silu (matmul enable=0xd → silu enable=0x18, ERDMA off) is proven (ork_npu_chain_gatesilu_i16). The mm→silu template explicitly zeroes 0x5038/0x5040.
  • 2-input SDP (ERDMA active) as a chained task HANGS (errno=110, self-heals). Confirmed for our per-channel scale AND per-element mul (ork_npu_chain_mm_perchan_i16, ORK_CHAIN_PE). Standalone the same op works (with or without ACT_RESET), so it's chain-specific.

Capture: ERDMA can chain (rules out a fundamental wall)

tools/re/models/build_fuse.py (Conv→Mul/Add, 2-input) compiled on the .239 Colima VM (rknn-toolkit2 2.3.2, OpFusing ran) → conv_mul.rknn, captured on-board (~/conv_mul.dump, regcmd_capture.so). 6 tasks: task[0]=conv (enable=0x1d), task[1]=the fused 2-input mul SDP (enable=0x18, 69 regs) — and it uses ERDMA: 0x5034=0x40000008 (ERDMA_DATA_MODE=1 per-element + DATA_SIZE=2), 0x5038=2nd operand. So chained-ERDMA is real. Templatized verbatim as REGCMD_MUL_F16_CHAIN (regcmd_ewmul.h).

Key result: the blocker is the MATMUL→2-input-SDP HANDOFF, not the SDP regcmd

Substituting the vendor's exact REGCMD_MUL_F16_CHAIN as our chained SDP task (patching only addresses/geometry/ERDMA-mode) STILL hangs after our matmul. Since that same regcmd works as task[1] after the vendor's conv (task[0], enable=0x1d), the problem is our first task: synth_i8 (enable=0xd) does not arm/hand-off to a following 2-input-SDP the way the vendor's conv does. (1-input silu is fine after our matmul — so it's specifically the 2nd-operand/ERDMA arming across the chain boundary.)

Open RE (roadmap)

  1. Capture + decode the vendor task[0] (conv) too. Diff its regcmd (esp. enable=0x1d vs our 0xd, and any ERDMA/next-task-arm registers) against synth_i8 to find what arms the following 2-input SDP.
  2. Decode REGCMD_MUL_F16_CHAIN's unknown registers. We use ~8 knowingly (0x4020 out, 0x5018/0x5038 operands, 0x5034 ERDMA_CFG, 0x4040/0x4060 BS/BN bypass=0x53, 0x4070 EW_CFG=0x108003c4, 0x4084/88 out-cvt, set_mul_geom geometry). Undecoded (used verbatim): 0x4010,0x404c,0x4050,0x4054, 0x4064,0x4068,0x406c,0x4074,0x4078,0x407c,0x4090–0x412c,0x40c0/c4, 0x5044,0x5068, and the trailer (0x0014 chain-amount / 0x0041,0x0081 descriptors). rocket_registers.h NAMES most — cross- reference each to the ISA Reference. RE debt: don't ship an opaque 146-word blob.
  3. Register naming belongs on the [regcmd ISA Reference]; the chained-ERDMA arming finding on NPU-Quirks.

Status

Everything the chain is made of is validated standalone on silicon (row_max, per-channel scale i8/f16/i16, exp, Σ, and the full transposed softmax coherent end-to-end). The single-submit chain is blocked ONLY on the matmul→2-input-SDP handoff above — a bounded, board-safe RE (capture+decode the conv task0), not a new capability. Board-hang caution: each chain miss = a ~60s timeout+soft-reset; use ORK_EW_TIMEOUT=1500, never brute-force.

Update — decoded the rocket-mapped registers (tools/re/decode_rocket.py)

Cross-referenced both vendor tasks against rocket_registers.h (decode_rocket.py names each reg + fields):

  • conv task[0] (working handoff): 0x4010 DATA_FORMAT=0x48000002 (PROC/IN/OUT_PRECISION=2=fp16), 0x4020 DST_BASE=0xfffef000 (writes fp16 to mem), 0x4040 BS_CFG=0x20150 (BS-ALU active ALGO=2), 0x4070 EW_CFG=0x383 (EW bypassed), 0x400c FEATURE_MODE=0x1e4, 0x5044 RDMA_FEATURE MRDMA_DISABLE=1.
  • mul task[1] (2-input SDP): DATA_FORMAT fp16, 0x4070 EW_CFG=0x108003c4 (EW_OP_TYPE=1 mul, EW_OP_SRC=1, EW_DATA_MODE=1, EDATA_SIZE=2), 0x5034 ERDMA=0x40000008, 0x4040/0x4060 BS/BN bypassed (0x53), FEATURE_MODE=0x1e5. => Hang cause hypothesis: DTYPE-PATH MISMATCH. Vendor chain is ALL-fp16 (fp16 conv-out -> fp16 SDP). Ours feeds int8-matmul -> int16 G -> the fp16-configured SDP; PROC_PRECISION=2(fp16) reading int16 bytes stalls the chained DMA. Consistent with 1-input int16 silu chaining fine but the fp16 2-input SDP on int16 data hanging. NEXT: match the all-fp16 path — matmul must output fp16 (DATA_FORMAT OUT_PRECISION=2), like the vendor conv, so the fp16 SDP reads fp16. (Needs a fp16-output matmul stage; the fp16 stream currently outputs fp32.)

RESOLVED — the chained 2-input SDP hang is FIXED (dtype-path match)

Confirmed on-silicon: matmul(fp16-out, set_f16_out) → fp16 per-channel SDP chains with rc=0, NO hang (was errno=110). The deepest M4 blocker is solved — a 2-input SDP (ERDMA) DOES run as a chained non-first task once the precision path matches (SDP is PROC_PRECISION=2 fp16; feeding it non-fp16 bytes stalled the chained DMA). Remaining for correct values: use a fp16-IN matmul (the int8→fp16 set_f16_out test path yields G=0 — its wedge-prone CVT). The real attention A·V is fp16-in, so that's the natural closing path + a matmul-out-vs-SDP-in layout (EWCUBEH) check. Chain-walk = CLOSED; value path = mechanical.

fp16-IN value close — fp16 matmul has NO 2-byte DPU writeout (HW datapath wall)

With the hang fixed, closing the CORRECT-value path needs the matmul to emit fp16 (int8→fp16 = zeros). Decoded the vendor conv task[0] fp16 output stage (conv_mul_full.dump → decode_rocket.py) and found the one thing set_f16_out was missing: 0x4084 DPU_OUT_CVT_SCALE = 0x00010001 (FP32TOFP16_EN, bit16) — the enable for the fp32→fp16 writeout conversion (set_f16_out wrote 0x1, no enable). synth's fp16 matmul template already matches the vendor on CNA fp16 (0x100c=0x20000120), CVT bypass (0x104c=0xb), 0x400c OUTPUT_MODE=2, 0x40c0 — so the true delta to fp16-out is just OUT_PRECISION + FP32TOFP16_EN.

But all three fp16-out configs HANG the fp16 matmul standalone (task_number=1, errno=110): the full vendor output stage, a no-geometry variant, and the 2-register minimal delta — while the SAME matmul with fp32-out runs fine. int8-in + FP32TOFP16_EN still yields zeros (int32 accumulator, no int32→fp16 CVT).

=> The fp16 matmul datapath (enable=0xd) has no working 2-byte fp16 DPU writeout. The vendor emits fp16 ONLY from the CONV datapath (enable=0x1d). This is the same class of limit as int8→fp16 fused is not a datapath: on-chip fp16 OUTPUT is constrained to the conv/fused path. The single-submit fp16 A·V-normalize bridge therefore needs either the conv front-end reproduced for a matmul-equivalent, or a chained SDP that reads fp32 G. The separate-submit attention normalize is already coherent and the chain is throughput-neutral, so this is an architectural nicety, not a perf lever. Harness preserved on feat/attn-primitives (ork_npu_chain_mm_perchan_f16, ork_npu_probe_f16_mm_f16out, set_f16_out_fp16in).