Benchmarks 2024 11 26 TFLM LLVM Os spike_rv32_min - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
eb0a3e2b0a7c57522928be39de95cd9f8c6dc636 - Spike PK :
fix-gcc14-rvv
- Spike :
Toolchains
-
RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Vector:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
-
LLVM/Clang:
clang version 18.1.8 (https://github.com/llvm/llvm-project.git 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)- Linker: lld (TODO)
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TFLM :
8eb6b23de4470d6a8da3131650d6a67514dfa130
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: llvm, Flags: -Os, Target: spike_rv32_min )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
39232601.0 ( 0.4x ) |
202416 ( 0.856 ) |
35992 ( 1.0 ) |
0 | TFLM | Reference | RV32IM | 0 | - |
33275950.0 ( 0.5x ) |
208396 ( 0.882 ) |
35992 ( 1.0 ) |
128 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
31483286.0 ( 0.5x ) |
208380 ( 0.882 ) |
35992 ( 1.0 ) |
256 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
208684 ( 0.883 ) |
35992 ( 1.0 ) |
512 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
30133659.0 ( 0.5x ) |
209024 ( 0.884 ) |
35992 ( 1.0 ) |
1024 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
30134089.0 ( 0.5x ) |
209228 ( 0.885 ) |
35992 ( 1.0 ) |
2048 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
30134050.0 ( 0.5x ) |
209432 ( 0.886 ) |
35992 ( 1.0 ) |
4096 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
15024169.0 ( Base ) |
236344 ( Base ) |
35992 ( Base ) |
0 | muRISCV-NN | Scalar | RV32IM | 0 | - |
14914746.0 ( 1.0x ) |
235156 ( 0.995 ) |
35992 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32IM | 0 | - |
6087854.0 ( 2.5x ) |
246468 ( 1.043 ) |
35992 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
5081221.0 ( 3.0x ) |
246416 ( 1.043 ) |
35992 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
4581047.0 ( 3.3x ) |
246552 ( 1.043 ) |
35992 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
4288945.0 ( 3.5x ) |
246776 ( 1.044 ) |
35992 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
4288852.0 ( 3.5x ) |
246872 ( 1.045 ) |
35992 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
4286501.0 ( 3.5x ) |
246968 ( 1.045 ) |
35992 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
4103590.0 ( 3.7x ) |
237520 ( 1.005 ) |
35992 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
2848612.0 ( 5.3x ) |
237412 ( 1.005 ) |
35992 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
2166149.0 ( 6.9x ) |
237408 ( 1.005 ) |
35992 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
2130388.0 ( 7.1x ) |
237460 ( 1.005 ) |
35992 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
2130809.0 ( 7.1x ) |
237460 ( 1.005 ) |
35992 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
2131134.0 ( 7.0x ) |
237460 ( 1.005 ) |
35992 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
6842130.0 ( 2.2x ) |
245108 ( 1.037 ) |
35992 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
5580208.0 ( 2.7x ) |
244920 ( 1.036 ) |
35992 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
5097433.0 ( 2.9x ) |
245056 ( 1.037 ) |
35992 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
245280 ( 1.038 ) |
35992 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
4824122.0 ( 3.1x ) |
245376 ( 1.038 ) |
35992 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
4824887.0 ( 3.1x ) |
245472 ( 1.039 ) |
35992 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
120988348.0 ( 0.5x ) |
245320 ( 0.916 ) |
68760 ( 1.0 ) |
0 | TFLM | Reference | RV32IM | 0 | - |
- ( ?x ) |
251104 ( 0.937 ) |
68760 ( 1.0 ) |
128 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
251176 ( 0.937 ) |
68760 ( 1.0 ) |
256 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
251680 ( 0.939 ) |
68760 ( 1.0 ) |
512 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
252140 ( 0.941 ) |
68760 ( 1.0 ) |
1024 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
252464 ( 0.942 ) |
68760 ( 1.0 ) |
2048 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
252788 ( 0.943 ) |
68760 ( 1.0 ) |
4096 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
56266438.0 ( Base ) |
267936 ( Base ) |
68760 ( Base ) |
0 | muRISCV-NN | Scalar | RV32IM | 0 | - |
72492194.0 ( 0.8x ) |
267072 ( 0.997 ) |
68760 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32IM | 0 | - |
- ( ?x ) |
278612 ( 1.04 ) |
68760 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
278616 ( 1.04 ) |
68760 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
278992 ( 1.041 ) |
68760 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
279360 ( 1.043 ) |
68760 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
279600 ( 1.044 ) |
68760 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
279840 ( 1.044 ) |
68760 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
15270545.0 ( 3.7x ) |
270228 ( 1.009 ) |
68760 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
9687783.0 ( 5.8x ) |
270128 ( 1.008 ) |
68760 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
7142289.0 ( 7.9x ) |
270124 ( 1.008 ) |
68760 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
5900248.0 ( 9.5x ) |
270176 ( 1.008 ) |
68760 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
4972641.0 ( 11.3x ) |
270176 ( 1.008 ) |
68760 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
4721620.0 ( 11.9x ) |
270176 ( 1.008 ) |
68760 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
- ( ?x ) |
277008 ( 1.034 ) |
68760 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
277012 ( 1.034 ) |
68760 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
277388 ( 1.035 ) |
68760 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
277756 ( 1.037 ) |
68760 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
277996 ( 1.038 ) |
68760 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
278236 ( 1.038 ) |
68760 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
2797784.0 ( 0.6x ) |
384944 ( 0.979 ) |
19336 ( 1.0 ) |
0 | TFLM | Reference | RV32IM | 0 | - |
- ( ?x ) |
387420 ( 0.985 ) |
19336 ( 1.0 ) |
128 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
387320 ( 0.985 ) |
19336 ( 1.0 ) |
256 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
387452 ( 0.985 ) |
19336 ( 1.0 ) |
512 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
387672 ( 0.986 ) |
19336 ( 1.0 ) |
1024 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
387756 ( 0.986 ) |
19336 ( 1.0 ) |
2048 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
387840 ( 0.986 ) |
19336 ( 1.0 ) |
4096 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
1700249.0 ( Base ) |
393160 ( Base ) |
19336 ( Base ) |
0 | muRISCV-NN | Scalar | RV32IM | 0 | - |
3052881.0 ( 0.6x ) |
393164 ( 1.0 ) |
19336 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32IM | 0 | - |
- ( ?x ) |
397540 ( 1.011 ) |
19336 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397412 ( 1.011 ) |
19336 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397488 ( 1.011 ) |
19336 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397672 ( 1.011 ) |
19336 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397720 ( 1.012 ) |
19336 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397768 ( 1.012 ) |
19336 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
1945586.0 ( 0.9x ) |
395372 ( 1.006 ) |
19336 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
1828906.0 ( 0.9x ) |
395276 ( 1.005 ) |
19336 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
1770566.0 ( 1.0x ) |
395284 ( 1.005 ) |
19336 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
1741720.0 ( 1.0x ) |
395344 ( 1.006 ) |
19336 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
1738089.0 ( 1.0x ) |
395344 ( 1.006 ) |
19336 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
1736233.0 ( 1.0x ) |
395344 ( 1.006 ) |
19336 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
- ( ?x ) |
397544 ( 1.011 ) |
19336 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397416 ( 1.011 ) |
19336 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397492 ( 1.011 ) |
19336 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397676 ( 1.011 ) |
19336 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397724 ( 1.012 ) |
19336 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
397772 ( 1.012 ) |
19336 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
103372570.0 ( 0.4x ) |
476056 ( 0.933 ) |
134296 ( 1.0 ) |
0 | TFLM | Reference | RV32IM | 0 | - |
- ( ?x ) |
482036 ( 0.945 ) |
134296 ( 1.0 ) |
128 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
482020 ( 0.945 ) |
134296 ( 1.0 ) |
256 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
482324 ( 0.946 ) |
134296 ( 1.0 ) |
512 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
482664 ( 0.946 ) |
134296 ( 1.0 ) |
1024 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
482868 ( 0.947 ) |
134296 ( 1.0 ) |
2048 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
483072 ( 0.947 ) |
134296 ( 1.0 ) |
4096 | TFLM | Reference | RV32IM_ZVE64X | 0 | Loop+SLP |
44946617.0 ( Base ) |
509984 ( Base ) |
134296 ( Base ) |
0 | muRISCV-NN | Scalar | RV32IM | 0 | - |
44831241.0 ( 1.0x ) |
508796 ( 0.998 ) |
134296 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32IM | 0 | - |
- ( ?x ) |
520108 ( 1.02 ) |
134296 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
520056 ( 1.02 ) |
134296 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
520192 ( 1.02 ) |
134296 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
520416 ( 1.02 ) |
134296 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
520512 ( 1.021 ) |
134296 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
520608 ( 1.021 ) |
134296 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32IM_ZVE64X | 0 | Loop+SLP |
13391092.0 ( 3.4x ) |
511160 ( 1.002 ) |
134296 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
10070110.0 ( 4.5x ) |
511052 ( 1.002 ) |
134296 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
8778682.0 ( 5.1x ) |
511048 ( 1.002 ) |
134296 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
8274774.0 ( 5.4x ) |
511100 ( 1.002 ) |
134296 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
8227414.0 ( 5.5x ) |
511100 ( 1.002 ) |
134296 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
8227733.0 ( 5.5x ) |
511100 ( 1.002 ) |
134296 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32IM_ZVE64X | 0 | - |
- ( ?x ) |
518748 ( 1.017 ) |
134296 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
518560 ( 1.017 ) |
134296 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
518696 ( 1.017 ) |
134296 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
518920 ( 1.018 ) |
134296 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
519016 ( 1.018 ) |
134296 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
- ( ?x ) |
519112 ( 1.018 ) |
134296 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32IM_ZVE64X | 0 | Loop+SLP |
Original data
Click here to download the raw files for this benchmark.