Hardware Chaining Debug (INT4/INT8)

Current Status

• Both ork_mm_run_chain_i4 and ork_mm_run_chain_i8 are failing during the RKNPU_SUBMIT ioctl phase.
• We observe errno=22 (Invalid argument) or errno=110 (Timeout) when attempting to submit multiple independent matmuls in a single ioctl call (task_number > 1).

Validation Methodology

During the debugging session, we adopted the following validation strategies:

1. Isolated Testing Executable (test_chain_i4_s1.c): Created a standalone test focused exclusively on the ork_mm_run_chain_i4 execution path for a single chaining task (S=1). This drastically reduced the noise and setup time compared to running the full test_matmul suite.
2. Overlap Guards (test_overlap_guards): Added active tests inside test_matmul.c to ensure that our chained implementation does not silently ignore or mishandle overlapping input (A) and output (C) buffers, which would violate the hardware execution constraints.
3. Target Device Validation: Synchronized source files locally and compiled/executed directly on the Rock 5B target board (10.3.0.236) using make test_chain_i4_s1 and sudo ./test_chain_i4_s1 to ensure actual kernel ioctl behavior was captured, as the macOS host cannot run these.
4. Ioctl Parameter Tracing: Intercepted the exact arguments passed to the DRM_IOCTL_RKNPU_SUBMIT call by inserting debug print statements directly into src/npu.c. We observed sub.flags, sub.task_number, sub.core_mask, t[0].regcfg_amount, and t[0].regcmd_addr just before kernel entry to confirm memory alignment and configuration correctness.

Recent Discoveries & Work Done

1. Unified Submission Logic: We restructured run_chain_i4 and run_chain_i8 to use a cleaner, unified approach for buffer creation (bcreate), synchronization (bsync), and ioctl submission.
2. Buffer Flags: We experimented with bcreate flags for the rknpu_task array, testing 0x403 vs 0x40b (IOMMU related), but the timeouts persist.
3. Register Configuration:
   • regcfg_amount is correctly set to 116 for i4 and 108 for i8.
   • synth_i4 and synth_i8 are correctly generating the regcmd buffers for each individual matmul task.
4. The probe_batch Warning: A previous note in src/npu.c (ork_npu_probe_batch) indicates that naive multi-task batching (task_number > 1) historically TIMES OUT. The NPU dispatches no tasks, and the kernel soft-resets. This suggests the hardware or DRM driver might not natively support cross-matmul batching via the task[] array in the way we are attempting.

Ideas for Next Steps

1. Investigate Subcore Tasks vs. Independent Tasks: The closed-source runtime (librkllmrt) handles large matmuls by splitting them into sub-tasks (e.g., 4 sub-tasks × 3 subcores = 12 tasks) within a single logical matmul. It might not support batching entirely independent matmuls via task_number > 1. We may need to investigate if chaining independent matmuls is actually possible, or if we need to emulate it.
2. Emulate Chaining via Sequential Submits: If true hardware-level chaining of independent matmuls is impossible, ork_mm_run_chain_* might need to be rewritten to issue sequential, single-task ioctl calls (similar to a loop of ork_mm_run_i8), potentially optimizing by preparing all regcmd buffers in advance to minimize CPU overhead between submits.
3. Kernel Logs (dmesg): When we resume, we should check dmesg on the Rock 5B board immediately after an errno=110 (Timeout) or errno=22 (Invalid argument) to see exactly what the rknpu DRM driver is complaining about. It often logs specific validation failures (e.g., memory alignment, invalid core masks, or bad regcmd jumps).
4. Compare with Multi-Core Split: Examine how ork_npu_mc_timing and the multi-core worker (i4_mcworker) successfully split a single task across cores. We might be missing a specific enable_mask, int_mask, or PC-jump configuration required to link the task[] entries together.