/* ⚠ CLOSED — the matmul-fused activation output is INT8-ONLY (2026-07-05 sweep). The output-precision is a
 * PREC field in reg 0x4010 (int8=bits00, int16=01, fp16=10; bit31=int32 which BYPASSES the CVT the LUT needs).
 * SWEPT PREC=1 (int16) across 3 output-stride configs (0x40c0/0x4050/0x4038) — ALL soft-reset the NPU + garbage.
 * int32 (0x800044e0) also wedges (CVT bypass vs LUT conflict). So the MATMUL+LUT program only supports int8
 * output; int16/fp16 output exists only in the STANDALONE silu program (0x50xx lane: int16=0x24004401,
 * fp16=0xa8000002 — different regcfg). No RKNN capture possible (RKNN never fuses activation into non-int8
 * matmul). CONCLUSION: higher-precision fused silu is not achievable in the matmul program. The ablation
 * (ORK_GATE_ABLATE — historical) proved int8 silu OUTPUT is the whole FFN-chain PPL gap, so the remaining route to parity
 * is UN-FUSED: int32 matmul -> standalone int16/fp16 silu op (loses the silu-free-on-NPU fusion, adds a submit).
 * Below is the sweep harness (env-configurable format regs), kept for the record; NOT called by the chain. ── */



/* ork_f16_mm_run_silu — fp16 gate matmul + fused SiLU with fp16→fp32 output (no int8 activation quant). The
 * "end-goal" precise on-NPU gate: recovers the full PPL gap the int8 silu output loses (ablation), at the cost
 * of the fp16 matmul (~3.3x int8, tools/f16_gate_bench) — a measured net-loss TODAY, so gated OFF, built out
 * for a future pipeline where it pays off. w = fp16 weight (ork_f16_mm_pack), A = fp16 [M,K], C = fp32 [M,N] silu.
 * K%32, N<=nmax. fp16 M-tile = orki_f16_mtile_n(K,M,N) = the 0x1040 ceiling, N-aware since r103 (was a stale
 * chunk=16; the real "latent bug" is only ABOVE that ceiling — bit-exact validated, see f16_mtile / silu_f16_check).
 * 0/ok, -1 wedge, -2 shape, -3 SoC. STATUS (2026-07-05): RUNS on-NPU (no wedge) AND now CALIBRATED accurate —
 * tools/silu_f16_calib cracked it to mean|err|~0.08 / max 0.75 over silu[-8,8] (~1%, on par with int8 silu).
 * CALIBRATION RECIPE (the fp16 program's index only spreads for NEGATIVE acc, so): (1) NEGATE the gate — pack
 * the gate weight *(-S)* so a positive real gate becomes a negative accumulator that spreads across the LUT;
 * negatives then clamp to ~0 (== silu(neg)). (2) scale S~24 (spreads [-8,8] over the ~513-entry negative half;
 * larger S over-spreads/clamps). (3) idx_off=0xffffc000, cfg4068=0x56391100, DEFAULT 0x4064 (0xffff7dc8
 * collapses it). (4) measure idx(acc) via a ramp LUT, build curve LUT[idx]=silu(-acc/S)/(R*out_scale). R
 * (output gain) = 0x4084; 0x4044/za has NO index effect. TODO to enable: a build_f16_silu_lut() baking (1)-(4)
 * + the negate/scale into the fp16 gate weight pack, then wire the gate. Kept gated OFF (fp16 mm ~3.3x int8). */


/* Resident-weight int8 matmul with int8-REQUANTIZED output (no activation): C = clamp_i8(round((A·W)*
 * mult/2^shift)) as int8 [M*N]. For keeping int8 intermediates on the NPU across a chained FFN inner
 * (the "up" projection feeding the EW-mul). Same resident full-K path + submit1 as the fused runs. */


/* RE/calibration: run ONE M=1 full-K int8 submit (no K-split) at (K,N) to probe this SoC's
 * single-submit K-tile ceiling (`0x1044`). Allocates its own buffers — does not touch resident
 * weights. Returns 0 if the submit completed (C[N] int32 valid), -1 if it wedged (K over the
 * per-op K-tile cap; recoverable — the next call's RKNPU_ACT_RESET clears it), -2 on bad dims. */
/* RE/calibration: run ONE full-K int8 submit at (M,K,N) in either ork's current synth_i8 M-tile
 * mode (mode=0) or the rkllm-captured M-tile mode (mode=1), and return the full C[M*N] int32 plus
 * the warm-submit time (us). rkllm-mode mirrors fused2.log: 0x1010=0x20 const (NOT 16*min(M+1,R)),
 * 0x1044=(K/64)*M, 0x107c=4*M, 0x1040=0xb1-0xf*(ceil(M/8)-1). Lets us test whether rkllm's larger
 * M-per-submit (M up to 36) is bit-exact on ork and whether it lifts effective TOPS. A[M*K] B[K*N]
 * row-major int8; C[M*N] int32. Returns 0/ok (C valid), -1 wedged, -2 bad dims. ISOLATED buffers. */


/* RE/validation for the PPU fused-output stage (step 1): run ONE full-K int8 matmul at (M,K,N) with
 * the int8-REQUANTIZED output stage (set_i8_out8) instead of int32, and return C[M*N] as int8. This
 * is the isolated bit-exact test bed for the fused path — the caller compares against the CPU model
 * out_i8 = clamp_i8((acc_i32 * mult) >> shift). Isolated buffers, does not touch resident weights.
 * A[M*K] B[K*N] row-major int8; C[M*N] int8 out. mult/shift = fixed-point requant (identity =
 * 0x4000,14). 0/ok (C valid), -1 wedged, -2 bad dims. See ork_ppu_fuse_enabled + set_i8_out8. */

/* PHASE 1 (#35 chained FFN) — RE crux: run one int8 matmul with the INT16-requantized output stage
 * (set_i16_out) and return C[M*N] as int16. Clone of probe_i8_out8 (weight [NT][KT][32][32], A->Af,
 * one submit) with a 2-byte output surface. Isolates whether the int16 output encoding produces the
 * correct VALUES (out_i16 = clamp_i16(round(acc_i32 * mult / 2^shift))); the caller compares vs the
 * CPU model. C is written as raw M*N*2 bytes (layout TBD — a varying-value pass follows to map it).
 * 0/ok, -1 wedged, -2 dims. */


/* DOORBELL PIPELINE PROFILER (Tier 11): measure the wall of `iters` SERIAL int8 matmuls, BLOCKING vs
 * NONBLOCK + DRAM-doorbell busy-poll. Blocking = submit waits (~130µs floor + compute)/op. Nonblock = submit
 * returns ~5µs, CPU busy-polls the output SENTINEL (DC CIVAC invalidate + read) until the NPU overwrites it =
 * op done, no sleep/wake. Same op (all-ones int8 -> every output = K) both ways; validates output == K. Calls
 * the raw ioctl directly with explicit flags (bypasses the env/sleep wrapper). 0/ok, <0 err. */

/* overlap_prof — Tier 11: does REAL CPU work in the shadow of an async NPU op stay FREE (overlap wall ~= max),
 * or does shared LPDDR4X bandwidth contention stretch it (wall -> npu+cpu)? This is the "zero-time router"
 * thesis (speculative/batched MoE) in one number. cpu_reps = # of 512x512 fp32 GEMVs run on the CPU between
 * the NONBLOCK submit and the doorbell poll (a stand-in for the routing math; 1MB matrix -> real DRAM traffic).
 * Fills npu_solo (submit+poll, no CPU work), cpu_solo (the GEMVs alone), overlap_wall (submit+CPU-work+poll). */


/* RE: STANDALONE fp16 matmul with FP16 output (set_f16_out, 0x4010=0x48000002 fp16-in) in the EWCUBEH atom-8
 * layout the chained SDP consumes — isolates "can the fp16 matmul emit fp16 G correctly" from the chain handoff.
 * A[M*K],B[K*N] fp16 bit patterns; out[M*N] fp16 read via EWCUBEH. task_number=1. 0/ok,-1 wedged,-2 dims. */

/* A1 (task #20): fp16 matmul with CONTIGUOUS fp16 output, consuming a PACKED resident ork_w (w->Bb) via the
 * PROVEN vendor fp16-out stage (orki_f16_set_out_fp16in, default contiguous). The point: a matmul's output is fp16
 * [M,N] directly — so it feeds an fp16 SDP op with NO f32->f16 host narrow between them, making the pair
 * ADJACENT (A2 can then keep the intermediate resident). Single-tile (M<=64, single-slice), single-core; it
 * runs its OWN submit (not the f32-out doorbell), so its SEQ_CLASS row is hw=0 (per-op SW dispatch). This is
 * the twin of ork_f16_npu_probe_mm_f16out but weight = w->Bb (resident) instead of a raw-B rebuild.
 * 0/ok, -1 submit-fail, -2 dims. */

/* ZERO-COPY STRIDED activation fp16 matmul — the densify-drop primitive for attention. A is logical [M][K] but
 * stored at row pitch `apitch` (apitch>=K, apitch%8==0) in a DEVICE (DMA) buffer; the NPU reads it DIRECTLY via
 * CNA LINE_STRIDE (0x107c=apitch/8), with NO host->Af gather. This is what a permuted-Q / KV-cache-view feeds:
 * the fork stages A once in a DMA buffer (KV-cache) and the matmul reads the strided view in place — no CPU
 * densify. Here the DMA buffer is allocated + filled internally to VALIDATE the zero-copy strided read against a
 * contiguous reference (out = A[:, :K]·B, contiguous fp16). A[M*apitch],B[K*N] fp16 patterns. 0/ok,-1,-2. */

/* SINGLE-SUBMIT fp16 matmul with per-channel scale FUSED into the output stage (BS-fold family): one task,
 * no separate SDP, no matmul-out<->SDP-in layout bridge. The EW-mul lane (0x50xx, spliced via orki_splice_ew_lane)
 * multiplies the ON-CHIP accumulator by a per-CHANNEL operand scale[N] (ERDMA per-channel 0x5034=0x08, operand
 * CONTIGUOUS [N]) before the fp16 writeout, so the output keeps the matmul's native CONTIGUOUS [M][N] layout
 * (proven 512/512). out[m][n] = (Σ_k A[m][k]B[k][n]) * scale[n]. A/B/scale/out fp16 bit patterns. Env-tunable
 * EW regs (ORK_F16EW_*) for on-board RE. 0/ok,-1 wedged,-2 dims,-3 SoC. */

/* RE/validation for the FUSED EW-mul output stage (step 3, SwiGLU dual-input): run a full-K int8 matmul
 * whose output stage int8-requantizes the accumulator AND multiplies it by a SECOND input G (= silu(gate)),
 * returning C[M*N] int8. This splices the 0x50xx second-DPU lane into the regcmd (synth_i8_ew) and submits
 * with regcfg_amount=126 (108 matmul + 18 second-lane). A[M*K] B[K*N] G[M*N] row-major int8.
 * FIRST-RUN CONTRACT: the 0x50xx dims/strides are the CAPTURED values (M=8,N=32) — validate at that shape
 * first; the exact multiply/scale semantics and the shape-dependent 0x50xx fields are resolved by board
 * matched-diff (see EWMUL_WIP.md). 0/ok (path executed, C valid), -1 wedged, -2 bad dims. */

/* PATH (b) bring-up: submit RKNN's captured EW-mul op (REGCMD_EWMUL) VERBATIM with ork's buffers, only
 * repointing the 6 buffer addresses. Its geometry is RKNN's own (unlike the synth_i8 overlay the HW
 * rejected), so this tests whether the templatized op EXECUTES on ork's submit path. Buffers mirror the
 * captured handle layout (input/weight/silu/output); contents are caller-provided (execution does not
 * depend on them). Returns 0/ok, -1 wedged. On ok, out[] gets the raw output buffer (Osz bytes).
 * in[Isz] wt[Wsz] gl[Gsz] are copied into the input/weight/silu buffers at their captured offsets. */

/* Path (b) MATMUL replay: submit the captured matmul-shaped EW-mul op (REGCMD_EWMUL_LIN, K=512/N=64/M=8)
 * with ork's buffers + ork's standard tile packing (B as [Nt][Kt][32][32], A linear). out = requant(A*B) ⊙ G,
 * G = silu(gate) int8 2nd-input. Fixed shape (the captured one) for now — used to read the multiply semantics
 * and validate vs CPU; generalization to arbitrary M/K/N is the next step. A[M*K] B[K*N] G[M*N] int8; C[M*N].
 * ORK_EW_S20=v writes uint32 v into the 0x5020 param region (2nd-input scale probe). 0/ok,-1 wedged,-2 dims. */

/* Standalone SDP element-wise MULTIPLY: submit REGCMD_MUL (captured pure-Mul op, no conv) with a,b,out patched.
 * out[i] = clamp_i8(round(a[i]*b[i]*gain) + bias), gain/bias baked from the captured op (0x4084/88/80).
 * a,b,out int8[n] (n<=4096). Reads a via SRDMA(0x5018), b via ERDMA(0x5038), writes out(0x4020). enable=0x18,
 * regcfg=69. This is the CLEAN on-NPU element-wise path (NVDLA standalone SDP layer, both operands from memory)
 * — sidesteps the conv-geometry coupling that blocked the fused-into-matmul approach. 0/ok,-1 wedged. */


/* fp16 element-wise MULTIPLY: out[m][n] = up[m][n] * silu[m][n] in fp16 on the NPU (standalone SDP fp16 op,
 * no requant). Marshals into the NVDLA fp16 feature cube (atom=8, 2-byte channels) internally. GENERALIZED to
 * arbitrary M,N (N a multiple of 8); rk3588-gated. up/silu/out are fp16 bit-patterns (ork_f16).
 * 0/ok,-1 wedged,-2 shape,-3 SoC. */

/* NEOX RoPE on the NPU (Qwen3 rope type 2). x[nrow][hd] fp16 row-major — nrow = (heads*tokens) flattened, each
 * row is one (head,token)'s head_dim vector; pos[nrow] = that row's token position. NEOX rotates pairs
 * (i, i+hd/2): out[i]=x[i]cosθ - x[i+hd/2]sinθ, out[i+hd/2]=x[i]sinθ + x[i+hd/2]cosθ, θ_i=pos*freq_base^(-2i/hd).
 * Composed on-NPU as x⊙COS + rot_half(x)⊙SIN — 2 ewmul + 1 add (COS/SIN tables + the half-swap built on CPU;
 * the transcendental-free EW math runs on the NPU). hd even, hd multiple of 8. 0/ok, <0 on primitive failure. */

/* int16 element-wise MULTIPLY: out[m][n] = clamp_i16(round(up[m][n]*silu[m][n] * mult/2^shift)) on the NPU
 * (standalone SDP int16 op). The w4a4 path's EW precision (ork's int4 matmul outputs int16). 2-byte operands,
 * NVDLA cube atom=8 (same layout as fp16). Symmetric zero-points. mult in 0..0x7fff. GENERALIZED to arbitrary
 * M,N (N a multiple of 8); rk3588-gated. 0/ok,-1 wedged,-2 shape,-3 SoC. */


/* On-NPU per-row MAX-REDUCE (int8): out[m] = max_n a[m*N+n], N%16, N<=8192. Batched pairwise-max TREE
 * using the SDP EW ALU in MAX mode (EW_ALU_ALGO=0 @ reg 0x4070, rocket_registers.h; NVDLA map_alu_op
 * MAX=0). Each level maxes channels [0,h) vs [h,2h) across ALL M rows in ONE submit: channel h sits at
 * cube offset (h/16)*M*16, so operand b = a + (h/16)*M*16 (valid since h stays %16 down the tree). Reduces
 * N->16 on-NPU (log2(N/16) submits, ping-pong buffers), then a 16-wide CPU tail. Reusable: softmax
 * stability, max-pool, top-k, clamp. 0/ok, <0 err. */


/* PURE-NPU per-channel-scaled fp16 matmul via a DIAGONAL second matmul (no SDP, no reshape, no CPU repack).
 * out = (A·B) · diag(scale): matmul1 A[M,K]·B[K,N] -> G[M,N] (contiguous fp16), then matmul2 G[M,N]·D[N,N] where
 * D=diag(scale) -> out[m][n]=Σ_k G[m][k]·D[k][n]=G[m][n]·scale[n]. The 2nd matmul reads G CONTIGUOUS as its
 * activation (synth's native activation layout) — the CNA reads it directly (the vendor-reshape discovery:
 * the CNA has a real feature LINE_STRIDE, unlike the DPU-RDMA). Cost: the diagonal adds O(M·N²) MACs (N×N
 * weight, mostly zero) — pure-NPU at the price of compute. A/B/scale/out fp16 bit patterns. 0/ok,<0. */

/* On-NPU PER-CHANNEL scale (fp16): out[m][n] = a[m][n] * b[n]. b[N] per-channel vector broadcast across all
 * M rows via ERDMA_DATA_MODE=0 (0x5034=0x08 = per-channel + DATA_SIZE TWO_BYTE for fp16). fp16 EW MUL,
 * quant-free (no gain/zero-points). N%8, N<=8192. b laid CONTIGUOUS [N] (fp16). The transposed-softmax
 * normalize (b=1/Σ per-query). 0/ok, <0 err. */

/* On-NPU PER-CHANNEL scale (int16): out[m][n] = clamp_i16(a[m][n]*b[n]*mult>>shift). b[N] per-channel
 * broadcast (ERDMA_DATA_MODE=0, 0x5034=0x08 for 2-byte). int16 chain-intermediate variant of the per-channel
 * scale (the M4 attention chain uses int16 between matmul and SDP, like the mm->silu chain). N%8. 0/ok, <0. */


/* Wall-#2 probe: a background thread that, mid-submit, hot-patches ONE descriptor far ahead in the treadmill
 * ring (outside the NPU's prefetch horizon) to a MARKER task, then flushes. If the NPU honors the live patch
 * (re-reads the descriptor from DRAM when it arrives), the marker's side effect appears. */

/* RE (WIP, RESHAPE_WIP.md): FULL-CHAIN REPLAY of the vendor gemm+reshape (task0-10) to validate the reshape
 * IN CONTEXT (task4 is a mid-chain op needing task1-3's pipeline state — standalone it saturates). Loads the
 * captured vendor IOVA image (gemm_mul_image.bin, 77824B, IB=0xfffed000), blanket single-delta rebases every
 * in-image reference (data/weights/chain-0x0010) to our buffer, replays task0..task10 as ONE hardware chain,
 * then hands back the GEMM output (@0xffff0000, contiguous [M][N]) and the RESHAPE output (@0xffff0a00, atom-8)
 * so the caller can verify reshape_out == atom8(gemm_out) IN-PLACE — no weight extraction / CPU ref needed.
 * M8/N64 captured geometry. Env ORK_RESHAPE_IMG=path. 0/ok, <0 err. */

/* RE PROBE (WIP, RESHAPE_WIP.md): submit the vendor fp16 contiguous->atom-8 RESHAPE base op (task4,
 * REGCMD_RESHAPE_F16) with a CONSTRUCTED permutation weight (64 all-1.0 entries = the captured N=64 pattern)
 * + our own contiguous [M][N] fp16 input, standalone (task_number=1, enable=0xd). Reads the output buffer RAW
 * so the caller can compare it to the atom-8 layout PCH16(m,n)=(n/8)*(M*16)+m*16+(n%8)*2. N=64 only (the
 * weight pattern + geometry are the captured M=8/N=64 op). Empirical step: learn what ONE reshape op does
 * (task4 is one M-tile of the vendor's M-tiled reshape) before chaining. 0/ok, <0 err. */

/* ── LOOPBACK requantizer (RE HARNESS — the loopback is NOT VIABLE; kept for the finding). Standalone SDP
 * reads INT32 (matmul accumulator) from DRAM, per-channel scale, requant -> int16.  The "loopback" idea was
 * to ROUTE AROUND the broken CNA->DPU requant-WDMA (narrow-output matmul does not self-complete) by doing
 * the 32->16 requant in a SEPARATE SDP pass (enable=0x18, DPU-RDMA — a different physical block): Pass-1 =
 * int8 matmul OUT_PRECISION=int32 (self-completes); Pass-2 = THIS op.  RESULT (2026-07-15, RK3588): the SDP
 * main feature RDMA (MRDMA) clamps to a 2-byte read: with the DPU in int32 mode it fetches HALF the bytes
 * (2E vs the 4E consumed), starving the pipeline mid-job -> WDMA terminal-count never reached -> errno=110.
 * Shape-INVARIANT at the default config (all 8 IN_PRECISION enums, M=1..32).
 * ★ OVER-FETCH FIX (2026-07-15, RK3588): the RDMA input dims (0x500c width / 0x5014 channel) are DECOUPLED
 * from the DPU output dims. INFLATING the RDMA element count (ORK_RQ_5014=2N-1) makes it fetch 4E bytes so
 * both terminal counters hit together -> the int32-input SDP SELF-COMPLETES (rc=0, ~30-80us). So int32 CAN be
 * read to completion — the "terminal silicon" wall was a fetch/consume byte-count DISAGREEMENT, fixable in SW.
 * ★ BUT the DPU still processes it as TWO int16 LANES, not a true int32: out[even 2k]=low16(a[k]),
 * out[odd 2k+1]=high16(a[k]) (spread across 2 output channels; out[n]<-a[n/2], odd=0 for a<2^16). So a true
 * int32-VALUE requant still isn't a single-datapath op (the low/high halves land in separate channels the
 * per-channel EW-mul can't recombine) — but for int16-range accumulators the even-channel (low-half) read is
 * usable. Full int32 needs recombining the lanes (open). Env: ORK_RQ_4010/MSTRIDE/5034/5044/5014/500C/DUMP;
 * argv "M N". See wiki Exp-2026-07-14 (narrow-output section). */

/* On-NPU PER-CHANNEL scale (int8): out[m][n] = clamp_i8( a[m][n] * b[n] * mult >> (shift-14) ). b is a
 * length-N per-channel vector broadcast across all M rows via the EW operand-b per-channel mode
 * (ERDMA_CFG 0x5034: ERDMA_DATA_MODE bits[31:30]=0, DATA_SIZE bits[3:2]=1 => reg 0x00000004; confirmed
 * on RK3588 — DATA_MODE=1 is per-element, =0 is per-channel broadcast). EW MUL unit; N%16, N<=8192.
 * Reusable: softmax normalize (b=1/Σ), LayerNorm/RMSNorm affine, per-channel requant. 0/ok, <0 err. */

/* RE PROBE: on-NPU PER-CHANNEL scale via the SDP BS (bias/scale) stage reading a per-channel vector from
 * memory. out[m][n] = a[m][n] * scale[n]  (scale broadcast across rows = NATIVE per-channel, no tiling).
 * Regs (rocket_registers.h — RK3588 offsets, NOT vanilla NVDLA): 0x501c RDMA_BRDMA_CFG (BRDMA_DATA_USE
 * bits[4:1]), 0x5020 RDMA_BS_BASE_ADDR (scale vector src); 0x4040 BS_CFG (BS_BYPASS b0 / BS_ALU_BYPASS b1
 * / BS_MUL_BYPASS b4 / BS_RELU_BYPASS b6), 0x4048 BS_MUL_CFG (BS_MUL_SRC b0 =MEM, BS_MUL_SHIFT_VALUE b[13:8]).
 * Config regs were env-overridable (ORK_BS_R40/R48/BRDMA — removed) to pin DATA_USE/enable on-board. int8. 0/ok. */

/* Public on-NPU element-wise ADD (int8): out[m*N+n] = clamp_i8(round( (a*a_scale + b*b_scale)/out_scale ))
 * via the 2-input SDP op with ALU=add. Decoded structure: out = clamp((a*mult_a + b*mult_b) >> (0x4088-14) + zo),
 * mult_a=0x4084, mult_b=0x4078 are Q-format scale ratios (a_scale/out_scale, b_scale/out_scale). Symmetric quant
 * (zero-points 0). EXACT for RESIDUAL add (a_scale==b_scale==out_scale => out=clamp_i8(a+b)) and power-of-2
 * scale ratios; ARBITRARY unequal scales are approximate (the wide b-scale field 0x4078 isn't fully decoded).
 * Residual connections use equal scales, so the exact case is the intended one.
 * in/out int8 [M*N], N%16==0; rk3588-gated. 0/ok,-1,-2,-3. */

/* On-NPU fp16 element-wise ADD (residual): out[m][n] = a[m][n] + b[m][n] in fp16 via the 2-input SDP ALU=add op
 * (REGCMD_ADD_F16, gain 1). NVDLA fp16 cube (atom-8, 2-byte). in/out fp16 [M*N], N%8==0. 0/ok,-1,-2,-3. */

/* On-NPU int16 element-wise ADD: out = clamp_i16(round((a*a_scale + b*b_scale)/out_scale)) via the 2-input SDP
 * ALU=add op (REGCMD_ADD_I16). Same requant structure as int8. Residual (equal scales) => clamp_i16(a+b), exact;
 * arbitrary unequal scales approximate. in/out int16 [M*N], N%8==0. 0/ok,-1,-2,-3. */


/* fp16 standalone activation-LUT op — RE probe. Applies the PPU LUT to a SINGLE fp16 memory input [M][N] via
 * REGCMD_SILU_STD_F16 (single-input, fp16 gain 0x00010001). Two submits: REGCMD_SILU_LUT (load) + this op.
 * The LUT data words are streamed via 0x4104 verbatim (encoding TBD by calibration — pass lut as the raw 16-bit
 * words the op consumes). in/out fp16 [M*N], N%8==0; rk3588-gated. 0/ok,-1 wedged,-2 shape,-3 SoC. */


/* ---- FORWARD-SOFTMAX REPLAY (RE) --------------------------------------------------------------------
 * Replay the captured vendor forward-softmax 9-task PC-chained graph VERBATIM at its capture geometry
 * (reduction N=64, 256 rows). Proves the activation->matmul asymmetry (NPU-Quirks) is resolved by
 * PC-chaining all 9 tasks in ONE submit: max -> broadcast -> x-max -> exp(LUT) -> Sum(=exp.ones, a MATMUL
 * right after the exp SDP op) -> 1/Sum -> exp*(1/Sum). Every register VALUE is vendor-validated; we only
 * rebase DATA addresses (0x4020/0x5018/0x5038/0x1070/0x1110) + the 0101:0x0010 chain to our buffers, so
 * hard-wedge risk is low (a bad address soft-times-out). `in`/`out` are raw 32768-byte buffer images (the
 * captured h4/h5); fill `in` uniformly and expect softmax=1/64 everywhere (layout-agnostic). Buffers:
 *   IN  [0xfffb6000,+0x8000)  OUT [0xfffae000,+0x8000)  SCR [0xfffbe000,+0x3a000)  WT [0xffffa280,+..)
 *   LUT scratch [0xffffab00,..) = task4's throwaway output (the exp LUT itself loads to DPU SRAM).
 * 0/ok, -1 wedge/submit-fail, -2 alloc, -3 SoC. */

/* int16 (w16a16i) standalone activation-LUT op — RE probe. Same requant-LUT math as the int8 op
 * (out=clamp_i16(R*LUT-interp(idx)+out_bias)) but int16 I/O (atom-8 2-byte cube) via REGCMD_SILU_STD_I16.
 * Two submits (LUT-load + op). Caller supplies scale regs + LUT. in/out int16 [M*N], N%8==0. 0/ok,-1,-2,-3. */

/* Replay an ASSEMBLED int16 LUT-op (from tools/re/assemble_op.c): stream `lut` via the LUT-loader, then run the
 * given `regcmd` VERBATIM (RKNN's matched index/scale params baked in) — patching only addresses + M/N geometry.
 * Bit-exact to RKNN, no index decode. in/out int16 [M*N], N%8==0. 0/ok,-1 wedged,-2 shape,-3 SoC. */
/* #38 RE: replay a CAPTURED int8 matmul regcmd (e.g. rkllm's, from tools/re/regcmd_capture) on ork's own
 * submit path, to TIME the foreign schedule on the same silicon. Data is garbage (memset) — TIMING ONLY,
 * output not checked. Allocates A[M*K]/B[K*N]/C[M*N*4] in the active domain, patches the A/B/C address regs
 * (0x1070/0x1110/0x4020 — the only addresses a plain INT8_MM_INT8_TO_INT32 regcmd references) to them,
 * submits single-core task_number=1 (chain-links ignored at n=1), warms once, times `iters`. rkllm's task
 * descriptor is identical to ork's (enable=0xd,int_mask=0x300,regcfg_amount=108 — verified from capture).
 * SAFETY: orki_mm_timeout_ms() so a bad regcmd self-terminates (soft, reboot-recoverable) rather than hanging. */
/* #39 A-layout MAPPER (Route A): recover the fold's A-read map — for each weight one-hot position Bpos[i]
 * (= ork_woff byte for logical (n0, k0_i)), which A-buffer BYTE OFFSET does the fold pull for each output
 * position? With a one-hot weight at (k0,n0), C[*][n0] = A_read(*,k0). We run 4 submits/k0 on ONE buffer set
 * (stable IOVA, wedge-safe): (P) presence A=1 -> the raw-C entries that are 1 mark this k0's M output slots;
 * (0,1,2) A[j]=(j>>7p)&0x7f base-128 digits -> offset = d0 | d1<<7 | d2<<14 at each slot. Fills, per k0 i:
 * rpos[i*M+t] = raw int32 index of the t-th output slot, aoff[i*M+t] = the A byte offset it read (t<cnt[i]).
 * Returns 0/ok. Offsets up to 2MB (fits the oversized A buffer). */

/* RE/validation for the FUSED SiLU output stage (step 2): run a full-K int8 matmul with SiLU applied
 * on-chip, returning C[M*N] as int8. TWO submits on the single-stream NPU: (1) the LUT-load program
 * (REGCMD_SILU_LUT, enable=0x18) streams the int16 silu curve into PPU LUT SRAM; (2) the matmul compute
 * (synth_i8 + set_i8_silu, enable=0x1d) reads that LUT in its output stage. The LUT persists in SRAM
 * between the two sequential submits. Isolated buffers. A[M*K] B[K*N] row-major int8; C[M*N] int8.
 * 0/ok (path executed, C valid), -1 wedged, -2 bad dims. FIRST-RUN status: replays the mm_silu capture
 * scale/LUT verbatim — proves the path EXECUTES + LUT-persists; per-scale generation is WIP. */
/* Fused-SiLU probe with the decoded knobs exposed (see set_i8_silu): r_mult/r_shift = the unified scale R
 * (0x4084/0x4088), out_bias = 0x4080, idx_off = 0x4110, cfg4068 = 0x4068. lut != NULL overrides the LUT
 * contents (streams the first `nlut` int16 values into the LUT-load's 0x4104 writes) — for the staircase/
 * const-LUT calibration harness; lut==NULL keeps the captured fixed silu curve. */



/* Constant SDP index params for the standalone activation-LUT op (from the RKNN SiLU capture). The op's
 * index math idx(in) depends ONLY on these (not on in/out scale), so one calibration serves every scale. */

/* Calibrate idx(in) for the standalone activation op ONCE per ctx: run a ramp LUT (LUT[i]=i-512) at R=0.5
 * (out=clamp_i8(0.5*(idx-512)) -> idx=2*out+512), sweeping all 256 int8 inputs. R=0.5 keeps out unclamped
 * across the full input range. Fills c->silu_idx[(uint8)in] (=-1 where saturated). 0/ok, -1 wedged. */

/* build a LUT curve at the calibrated indices for f(v*in_scale)/out_scale (R=1); interp gaps, hold ends */
/* build the SDP activation LUT for f((vv-bias)*in_scale)/out_scale over int8 index vv. bias is a scalar shift
 * in int8-input units (0 for the plain curve). Used by softmax's scalar GLOBAL-max subtract: exp((x-max)*sc)
 * keeps the argument <=0 (output in (0,1]) so int8 exp never overflows — a scalar bias is bakeable into the
 * LUT (unlike a per-row max, which would need the dead per-channel-add), and the constant cancels in P=e/Sum. */

/* Generic int8 pointwise activation via the standalone SDP LUT op: out = clamp_i8(round( f((in-bias)*in_scale)/out_scale )).
 * The op is activation-agnostic (the LUT contents define f); calibrate idx once, build f's curve, run. bias is a
 * scalar shift in int8-input units (0 for plain). */

/* Public on-NPU SiLU (int8): out[m*N+n] = clamp_i8(round( silu(in[m][n]*in_scale) / out_scale )) via the
 * standalone SDP activation-LUT op. Lazily calibrates idx(in) once per ctx, builds the silu curve for the
 * requested (in_scale,out_scale), loads it, and runs the op (2 submits). in/out int8 [M*N], N%16==0.
 * rk3588-gated. 0/ok, -1 wedged, -2 bad shape, -3 non-rk3588. */
/* On-NPU GELU (int8): same standalone activation-LUT op, GELU curve. Bit-exact-class like SiLU. */
/* On-NPU rsqrt (int8) — RMSNorm building block: out = clamp_i8(round( rsqrt(in*in_scale)/out_scale )). */
/* On-NPU exp (int8) — softmax building block: out = clamp_i8(round( exp(in*in_scale)/out_scale )). */
/* On-NPU exp with a scalar GLOBAL-max subtract baked into the LUT: out = clamp_i8(round( exp((in-max)*in_scale)/out_scale )).
 * The softmax numerator with numerically-stable max-subtract, WITHOUT a per-row op: max is the scalar global max
 * (>= every row max, so every argument (in-max)<=0 => exp in (0,1], no int8 overflow), and the constant cancels
 * in P=e/Sum. `max` is in int8-input units (the int8 score value). Direct only for now (Path B TODO). */

/* int16 index params = RKNN's CAPTURED int16 SiLU index params (from REGCMD_SILU_STD_I16). Their gain (~0.008)
 * maps the FULL int16 input range onto a ~525-wide LUT-index band centred at ~510 — the same regime RKNN uses,
 * so the idx is densely + smoothly sampled and place-and-interpolate reproduces silu well (unlike int8's params
 * which give gain~2 and only span |in|<256). We keep these index params and only override R->1 / out_bias->0,
 * then build our own curve for the caller's (in_scale,out_scale) — mirroring the working int8 path. */

/* Calibrate the int16 idx map once per ctx at RKNN's index params, AT R=1 (== the run R, so the measured idx
 * matches the run exactly — no R-dependent idx shift). Ramp LUT (interpolated exactly) -> out=idx-512 ->
 * idx=out+512 (integer). Dense sampling (step 16) resolves the idx transitions for accurate inversion.
 * silu_idx16[s] = measured integer LUT index for q_in=-32768+s*16; INT16_MIN if saturated. */

/* PHASE 0 (#35 chained-FFN): a HETEROGENEOUS 2-task PC-chain in ONE submit — task0 = a tiny int8 matmul
 * (enable 0xd), task1 = the int16 silu SDP LUT-op (enable 0x18) — to prove the hardware walks the
 * CNA/DPU(matmul)->pure-SDP(silu) transition WITHOUT a per-op host round-trip/reset (the FFN-chain-critical
 * transition). Data handoff is NOT wired yet: the matmul writes a scratch (verifies task0 ran via Cd), the
 * silu reads a real int16 `in` (verifies task1 ran via `out`). Both correct + rc=0 => the heterogeneous
 * chain walked. Descriptor format mirrors the mcworker PC-chain (rc[216..219], 0x0101 next-addr/regamt).
 * in/out int16 [M*N], N%8==0. 0/ok, -1 wedge/timeout, -2 shape/alloc, -3 non-rk3588. */

/* (b) MIXED-PRECISION CHAIN probe: chain a FP16 matmul task (synth) + an INT16 silu-SDP task in ONE submit
 * and validate BOTH — the exact fp16-matmul + int16-elementwise mix the fused SSD scan needs. Confirms that
 * fp16 and int16 tasks coexist in one PC-chain (precision is per-task regcmd on the shared 2-byte datapath;
 * no inter-task reset). Mirrors chain_mm_silu_i16 but task0 is fp16 (all-1s 32x32 -> C=32.0) instead of int8.
 * Sets *mm_ok (fp16 matmul C≈32 in-chain), *silu_ok (int16 silu vs CPU). 0/ok,-1 fail,-2 bad,-3 SoC. */



/* fp16-IN chained matmul -> per-channel SDP (the attention A·V normalize path, all-fp16 like the vendor conv->mul).
 * Closes the single-submit chain with CORRECT values: fp16 matmul (synth, weight tile [N/16][K/32][16][32], fp16
 * activation) -> fp16-out G (set_f16_out with 0x4010=0x48000002 fp16-in) -> the vendor REGCMD_MUL_F16_CHAIN 2-input
 * SDP scales G per-channel by scale[N]. A[M*K],B[K*N],scale[N],out[M*N] are fp16 bit patterns (uint16). 0/ok,-1,-2,-3. */

/* (B') fp16 cross-slice DRAIN-GAP probe. Chains TWO wide fp16 matmuls with DISTINCT weight buffers (W0,W1) —
 * reproducing the cross-buffer base-latch condition — into ONE PC-chain submit, optionally with an identity
 * mul_perchan_f16 GAP between them (REGCMD_MUL_F16_CHAIN, enable 0x18, 69 regs, middle desc_slot=138 per the
 * regcfg*2 convention). The gap is a pure time-filler on a DUMMY fp16 scratch (identity scale) — its only job is
 * to idle the weight-CDMA so W0's fetch drains before W1's base latches. Both matmuls use the fp16-out stage
 * (orki_f16_set_out_fp16in — the config with the PROVEN mm->perchan HW edge). Returns the submit rc (nonzero=wedge);
 * *nz0 / *nz1 = count of nonzero output words per matmul (coarse "did it compute" sanity). rk3588-gated. */

/* PER-CORE-FD concurrency probe. Runs a 3-core fp16 N-column-split matmul where EACH core submits on its OWN
 * fresh DRM fd (a separate open of the card), not the shared c->fd — to test whether per-core-fd isolation
 * changes the concurrent-fetch "wedge". Core i owns columns [n0,n0+Ncol), n0=i*Ncol, Ncol=N/cores; it computes
 * ALL M rows of its column slice as an independent fp16 matmul (fp16-out, converted to fp32 into Cout).
 *   mode 0: each core gets its OWN weight copy (bcreate on its own fd), tiled from B's column slice.
 *   mode 1: ONE shared dma-heap weight (full K*N, tiled once) imported into every core's fd (bimport +
 *           bimport_fd); each core points its weight addr at the byte offset of its column-tile block.
 * Constraint set (kept simple): Sk=1 (single K, no host accumulate), Sn=1 (N<=nmax), M<=64, K%32==0, N%16==0,
 * and N%(cores*16)==0 so each core's Ncol is a multiple of 16 (a clean col-tile boundary). *us = wall time
 * around the concurrent submit region. Returns 0 on a completed run — INCLUDING a wedge, in which case Cout
 * shows the dropped/garbage columns (that IS the signal); <0 only on genuine setup failure (-2 bad shape,
 * -1 open/alloc). See tools/percore_fd_probe.c. */

/* Public on-NPU SiLU (int16 / w16a16i): out = clamp_i16(round( silu(in*in_scale)/out_scale )) via the standalone
 * int16 activation-LUT op, using RKNN's captured index params (full-range coverage). Builds the LUT the way RKNN
 * does — for each integer LUT index k it finds the q_in whose idx==k (from the dense R=1 calibration transitions)
 * and samples silu there, so the op's interpolation lands on silu at the exact grid. RKNN-class accuracy.
 * in/out int16 [M*N], N%8==0. rk3588-gated. 0/ok,-1 wedged,-2 bad shape,-3 SoC. */
/* On-NPU GELU (int16 / w16a16i): same activation-LUT op, GELU curve. RKNN-class accuracy like SiLU int16. */
/* On-NPU rsqrt (int16) — RMSNorm building block. RKNN-class accuracy. */
/* On-NPU exp (int16) — softmax building block. RKNN-class accuracy. */

/* RE: does batching tasks per ioctl amortize the RKNPU_SUBMIT round-trip floor? Runs `ntask`
 * identical small int8 matmuls (single core) as (a) ntask separate task_number=1 ioctls vs (b) ONE
 * ioctl with task_number=ntask. Returns 0/ok, -1 wedge, -2 bad dims (K%32, N%32, 1<=ntask<=32).
 * FINDING (2026-06-13): the batched path (b) TIMES OUT (`task counter: 0x0` — NPU dispatches no tasks;
 * kernel soft-resets + recovers). The naive task[]/subcore config doesn't drive multi-task execution.
 * AND it's moot for cross-matmul batching: the closed runtime's captured 12-task submit is ONE
 * matmul's program (4 sub-tasks × 3 subcores), NOT multiple matmuls batched — so librkllmrt also does
 * ~1 submit/matmul and pays the same per-matmul submit floor (~11 tok/s on 1.7B, which ork-driver
 * matched). The floor is inherent; cross-matmul task-batching is not the reference's mechanism nor
 * the lever. See tools/batch_probe.c. */

/* RE: probe in-place K-slicing of a FULL-K weight buffer (for a single-layout decode+prefill).
 * Packs B[Kfull,N] fp16 in full-K tile layout, then runs ONE M=1 submit over k in [0,Kp) reading
 * from that buffer — i.e. the op processes Kp passes but the weights are laid out for Kfull. With
 * no override the per-N-tile stride is Kp's (N-tile 0 correct, 1+ wrong); pass reg/val overrides
 * (e.g. 0x1044, 0x1034, 0x1030 set to their full-K values) to hunt the stride register that makes
 * all N-tiles correct. C[N] = sum_{k<Kp} A[k]*B[k][n] if slicing is right. nov<=4. Returns 0/ok. */

/* RE: probe W4A4 (int4 A x int4 B -> int16 C) using the captured REGCMD_I4 (M=4, the capture's M).
 * A[M*K], B[K*N] hold int4 values as int8 in [-8,7]. `blayout`/`alayout` select candidate native
 * tile packings (2 int4/byte); the regcmd is correct (captured) so a layout combo that matches the
 * CPU reference reveals the native tile order. C[M*N] int16 = sum_k A[m][k]*B[k][n]. `nov`/`ovr_*`
 * patch extra CNA regs from the tool. Returns 0 ok, -1 wedge/abort, -2 bad dims. Single task[0]
 * submit — if the 12-task W4A4 program needs the other tasks (A-quant/reorder), this is wrong and a
 * multi-task path is needed (see ROADMAP). Layouts: 0=K-contig lo/hi, 1=K-contig hi/lo, 2=N-lane. */

/* RK3588/3576 int4 native layouts (DOCUMENTED in rknn_matmul_api.h, not guessed):
 *   A: (K/32, M, 32)        elem[kt][m][kk] = A[m][kt*32+kk]
 *   B: (N/64, K/32, 64, 32) elem[nt][kt][nl][kk] = B[kt*32+kk][nt*64+nl]   (B row-major [K][N])
 * 2 int4 packed per byte; `nib` toggles which of the two consecutive elements is the high nibble. */

/* RE/calibration: ONE multi-M int4 submit (mc=M) — the M-scheduler experiment for Tier 4b. A is laid
 * (K/32,M,32) native; B native (N/64,K/32,64,32); the regcmd's M-count regs are set to M (synth_i4
 * mc=M). Copies the RAW int16 output buffer (M*N int16, NO de-tile) to raw so the caller can deduce
 * the multi-M C layout against a CPU reference. Returns 0 if the submit completed, -1 wedged, -2 dims.
 * Single N-slice (N<=nmax), single core. See tools/i4_multim_probe.c. */



/* .orkpack compat token = library MAJOR version (atoi stops at the first '.'): a format-breaking change
 * requires a major bump, while minor/patch stay backward-compatible. See ork_npu.h. */

/* Max M rows a single full-K int8 submit handles at this K (mirrors orki_run()'s M>1 Bf tiling, npu.c
 * "Tier 1c-ii"). Each chain link is ONE full-K submit, so a task's M must not exceed this — else the
 * caller must split the task into M-tiles. Guards against wedging the (shared) NPU on an oversized mc. */

/* Optional fused-SiLU output stage for ONE task in a chain (the gate): that task's regcmd gets set_i8_silu
 * (int8 silu(gate) output instead of int32), and the silu LUT is streamed to SDP SRAM once before the chain.
 * Reuses run_chain_i8's proven warm/buffer/submit machinery. NULL = plain int32 chain (original behavior). */
/* Per-task op kind for a heterogeneous chain (the general OPTION-B path). SDP tasks (SILU/EWMUL) read prior
 * tasks' OUTPUT buffers by index (in0/in1), aliased -- the vendor matmul->SDP pattern. Matmul tasks emit int32
 * (MM32) or plain int8 (MM8, set_i8_out8 requant mult/shift) -- use MM8 when the output feeds an SDP task. */

/* ops     = per-task op array (the general path; NULL -> legacy task/sdp_task below).
 * task     = fused gate*silu task (set_i8_silu on the matmul output) -- KNOWN to wedge in a chain, kept for A/B.
 * sdp_task = single standalone int8 silu-SDP op reading the PREVIOUS task's output (legacy 2-task path). -1=off. */


/* ORKD coalesced SwiGLU FFN against 3 resident (is_orkd) weights — one ORKD_FFN round-trip, one on-NPU
 * chain submit (gate->silu->up->glu->down), intermediates never leave the NPU. Thin wrapper over the
 * orkd_ffn_i8 client; the caller owns the per-stage requant (mult/shift) + silu (in_scale,out_scale). */

/* ORKD fused attention core against 3 resident (is_orkd) weights (K^T[Kp,Nk], ones[Nk,32], V[Nk,dv]) — one
 * ORKD_ATTN round-trip, one on-NPU chain (QK^T->exp->reduce,e.V), e never leaves the NPU. Sigma[Nq*32] +
 * av[Nq*dv] returned; caller normalizes attn=av/Sigma. Thin wrapper over the orkd_attn_i8 client. */

/* Transport-transparent whole-layer op. c->daemon set => forward as one ORKD_LAYER round-trip; else run locally
 * (the orkd daemon's handle_layer lands here on its own direct ctx). See the header for the compute contract. */



/* GENERAL heterogeneous FFN chain: per-task ops[] (OP_MM32/OP_MM8/OP_SILU/OP_EWMUL); SDP tasks read prior
 * outputs by index (ops[i].in0/in1), aliased. The silu LUT for (in_scale,out_scale) is built internally.
 * Chains e.g. [gate(MM8) -> silu(SDP,in0=gate) -> up(MM8) -> glu(EWMUL,in0=silu,in1=up) -> down(MM32)] in ONE
 * submit. tasks[i].C receives that op's output (int8 for MM8/SDP, int32 for MM32). Single M-tile per task.
 * 0/ok,-1 wedge,-2 dims,-3 SoC. */
/* Same heterogeneous chain, but the SDP activation task (kind 2) applies EXP instead of SiLU — the HW-chained
 * softmax numerator: [QK^T(1) -> exp(2, in0=0) -> reduce(0, reads exp)] as ONE submit, e kept on-chip. Scores
 * must be <=0 (post-max domain) so exp in (0,1] fits int8. Only differs from _ffn by the curve fn. 0/ok,-1,-2,-3. */
/* Fused-exp chain, but the exp bakes in a SCALAR max-subtract: e = exp((score-max_bias)*in_scale)/out_scale.
 * max_bias=0 => plain exp (scores must already be <=0). A max_bias >= every real score keeps all args <=0 so the
 * int8 exp never saturates, and the constant cancels in the softmax normalize av/Sigma (registry: exp_biased_probe,
 * scalar global-max-biased exp_i8). This is what makes the fused attention core correct on REAL (positive) QK^T
 * scores without a live per-query max. See ork_i8_npu_exp_biased / orki_silu_build_curve_biased.
 * (Defined before the plain wrapper below: the standalone Makefile build does not pull ork_npu.h into this TU.) */


/* Run `nchains` fused exp chains round-robin across the cores (concurrent). chains[i] = that chain's S[i]-task
 * array; ops = the shared op graph; (in_scale,out_scale) the shared exp requant. Returns 0/ok, <0 err (first
 * failing chain's code). PRECONDITION: cores must be WARM (a prior matmul on each) — a cold core's first submit
 * wedges; a chain-only caller should warm via a multi-core matmul first (see chainrr_conc_probe). Local NPU only. */