Benchmarks 2024 11 26 TFLM LLVM Os spike_rv64 - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
eb0a3e2b0a7c57522928be39de95cd9f8c6dc636 - Spike PK :
fix-gcc14-rvv
- Spike :
Toolchains
-
RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Vector:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
-
LLVM/Clang:
clang version 18.1.8 (https://github.com/llvm/llvm-project.git 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)- Linker: lld (TODO)
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TFLM :
8eb6b23de4470d6a8da3131650d6a67514dfa130
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: llvm, Flags: -Os, Target: spike_rv64 )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
39715465.0 ( 0.4x ) |
145894 ( 0.867 ) |
38312 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
31995165.0 ( 0.5x ) |
151022 ( 0.898 ) |
38320 ( 1.0 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
30079482.0 ( 0.5x ) |
150948 ( 0.897 ) |
38320 ( 1.0 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
29113993.0 ( 0.5x ) |
151156 ( 0.899 ) |
38320 ( 1.0 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
28630598.0 ( 0.5x ) |
151466 ( 0.9 ) |
38320 ( 1.0 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
28630607.0 ( 0.5x ) |
151670 ( 0.902 ) |
38320 ( 1.0 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
28644647.0 ( 0.5x ) |
173666 ( 1.032 ) |
38320 ( 1.0 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
14708091.0 ( Base ) |
168210 ( Base ) |
38312 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
15078549.0 ( 1.0x ) |
167462 ( 0.996 ) |
38312 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
5820243.0 ( 2.5x ) |
176044 ( 1.047 ) |
38320 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
4782120.0 ( 3.1x ) |
175966 ( 1.046 ) |
38320 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
4265038.0 ( 3.4x ) |
175990 ( 1.046 ) |
38320 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
3964941.0 ( 3.7x ) |
176216 ( 1.048 ) |
38320 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
3964944.0 ( 3.7x ) |
176310 ( 1.048 ) |
38320 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
3970551.0 ( 3.7x ) |
188674 ( 1.122 ) |
38320 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
4074938.0 ( 3.6x ) |
169684 ( 1.009 ) |
38312 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2819922.0 ( 5.2x ) |
169542 ( 1.008 ) |
38312 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2142586.0 ( 6.9x ) |
169414 ( 1.007 ) |
38312 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2103455.0 ( 7.0x ) |
169444 ( 1.007 ) |
38312 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2103455.0 ( 7.0x ) |
169444 ( 1.007 ) |
38312 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2105320.0 ( 7.0x ) |
169444 ( 1.007 ) |
38312 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
6910795.0 ( 2.1x ) |
174910 ( 1.04 ) |
38320 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
5905744.0 ( 2.5x ) |
174832 ( 1.039 ) |
38320 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
5403333.0 ( 2.7x ) |
174856 ( 1.04 ) |
38320 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
5121769.0 ( 2.9x ) |
175082 ( 1.041 ) |
38320 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
5116646.0 ( 2.9x ) |
175176 ( 1.041 ) |
38320 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
5127377.0 ( 2.9x ) |
187540 ( 1.115 ) |
38320 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
121110788.0 ( 0.5x ) |
187306 ( 0.928 ) |
71080 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
57760563.0 ( 1.0x ) |
193368 ( 0.958 ) |
71088 ( 1.0 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
47307258.0 ( 1.2x ) |
193366 ( 0.958 ) |
71088 ( 1.0 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
44797070.0 ( 1.2x ) |
193794 ( 0.96 ) |
71088 ( 1.0 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
44334667.0 ( 1.3x ) |
194232 ( 0.962 ) |
71088 ( 1.0 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
44324043.0 ( 1.3x ) |
194556 ( 0.964 ) |
71088 ( 1.0 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
44344333.0 ( 1.3x ) |
216082 ( 1.07 ) |
71088 ( 1.0 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
55734444.0 ( Base ) |
201896 ( Base ) |
71080 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
71563966.0 ( 0.8x ) |
201358 ( 0.997 ) |
71080 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
25739743.0 ( 2.2x ) |
210808 ( 1.044 ) |
71088 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17776724.0 ( 3.1x ) |
210736 ( 1.044 ) |
71088 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
14051138.0 ( 4.0x ) |
211024 ( 1.045 ) |
71088 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
12355748.0 ( 4.5x ) |
211402 ( 1.047 ) |
71088 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
11526338.0 ( 4.8x ) |
211640 ( 1.048 ) |
71088 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
10900009.0 ( 5.1x ) |
228720 ( 1.133 ) |
71088 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
15246284.0 ( 3.7x ) |
204114 ( 1.011 ) |
71080 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
9663084.0 ( 5.8x ) |
203982 ( 1.01 ) |
71080 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
7117244.0 ( 7.8x ) |
203854 ( 1.01 ) |
71080 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
5876981.0 ( 9.5x ) |
203884 ( 1.01 ) |
71080 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
4948717.0 ( 11.3x ) |
203884 ( 1.01 ) |
71080 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
4701490.0 ( 11.9x ) |
203884 ( 1.01 ) |
71080 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
19843671.0 ( 2.8x ) |
209586 ( 1.038 ) |
71088 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
13551429.0 ( 4.1x ) |
209514 ( 1.038 ) |
71088 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
10601906.0 ( 5.3x ) |
209802 ( 1.039 ) |
71088 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
9260268.0 ( 6.0x ) |
210180 ( 1.041 ) |
71088 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
8610061.0 ( 6.5x ) |
210418 ( 1.042 ) |
71088 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
8124607.0 ( 6.9x ) |
227498 ( 1.127 ) |
71088 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
2781498.0 ( 0.6x ) |
340870 ( 0.984 ) |
21448 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
957732.0 ( 1.7x ) |
343230 ( 0.99 ) |
21448 ( 1.0 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
694594.0 ( 2.4x ) |
343080 ( 0.99 ) |
21448 ( 1.0 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
563040.0 ( 2.9x ) |
343094 ( 0.99 ) |
21448 ( 1.0 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
497278.0 ( 3.3x ) |
343308 ( 0.991 ) |
21448 ( 1.0 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
459286.0 ( 3.6x ) |
343392 ( 0.991 ) |
21448 ( 1.0 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
467540.0 ( 3.5x ) |
350760 ( 1.012 ) |
21448 ( 1.0 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
1632483.0 ( Base ) |
346574 ( Base ) |
21448 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
2978070.0 ( 0.5x ) |
346578 ( 1.0 ) |
21448 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
593018.0 ( 2.8x ) |
350270 ( 1.011 ) |
21448 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
475576.0 ( 3.4x ) |
350096 ( 1.01 ) |
21448 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
416854.0 ( 3.9x ) |
350050 ( 1.01 ) |
21448 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
387484.0 ( 4.2x ) |
350228 ( 1.011 ) |
21448 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
372818.0 ( 4.4x ) |
350276 ( 1.011 ) |
21448 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
376094.0 ( 4.3x ) |
355450 ( 1.026 ) |
21448 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
1935621.0 ( 0.8x ) |
348202 ( 1.005 ) |
21448 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
1817253.0 ( 0.9x ) |
348096 ( 1.004 ) |
21448 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
1758068.0 ( 0.9x ) |
347974 ( 1.004 ) |
21448 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
1728813.0 ( 0.9x ) |
348004 ( 1.004 ) |
21448 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
1725129.0 ( 0.9x ) |
348004 ( 1.004 ) |
21448 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
1723245.0 ( 0.9x ) |
348004 ( 1.004 ) |
21448 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
992695.0 ( 1.6x ) |
350274 ( 1.011 ) |
21448 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
694330.0 ( 2.4x ) |
350100 ( 1.01 ) |
21448 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
545142.0 ( 3.0x ) |
350054 ( 1.01 ) |
21448 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
470546.0 ( 3.5x ) |
350232 ( 1.011 ) |
21448 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
433245.0 ( 3.8x ) |
350280 ( 1.011 ) |
21448 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
433723.0 ( 3.8x ) |
355454 ( 1.026 ) |
21448 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
104294920.0 ( 0.4x ) |
419536 ( 0.949 ) |
136616 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
67375214.0 ( 0.7x ) |
424664 ( 0.961 ) |
136624 ( 1.0 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
61850250.0 ( 0.7x ) |
424590 ( 0.961 ) |
136624 ( 1.0 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
59218368.0 ( 0.7x ) |
424798 ( 0.961 ) |
136624 ( 1.0 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
58109096.0 ( 0.8x ) |
425108 ( 0.962 ) |
136624 ( 1.0 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
57659751.0 ( 0.8x ) |
425312 ( 0.963 ) |
136624 ( 1.0 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
57638766.0 ( 0.8x ) |
447076 ( 1.012 ) |
136624 ( 1.0 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
43982864.0 ( Base ) |
441852 ( Base ) |
136616 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
45000202.0 ( 1.0x ) |
441104 ( 0.998 ) |
136616 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
18369396.0 ( 2.4x ) |
449686 ( 1.018 ) |
136624 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
15392478.0 ( 2.9x ) |
449608 ( 1.018 ) |
136624 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13968194.0 ( 3.1x ) |
449632 ( 1.018 ) |
136624 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13442124.0 ( 3.3x ) |
449858 ( 1.018 ) |
136624 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13226523.0 ( 3.3x ) |
449952 ( 1.018 ) |
136624 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13213856.0 ( 3.3x ) |
462084 ( 1.046 ) |
136624 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13376084.0 ( 3.3x ) |
443326 ( 1.003 ) |
136616 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
10055028.0 ( 4.4x ) |
443184 ( 1.003 ) |
136616 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
8763584.0 ( 5.0x ) |
443056 ( 1.003 ) |
136616 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
8261445.0 ( 5.3x ) |
443086 ( 1.003 ) |
136616 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
8213670.0 ( 5.4x ) |
443086 ( 1.003 ) |
136616 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
8215535.0 ( 5.4x ) |
443086 ( 1.003 ) |
136616 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
21985474.0 ( 2.0x ) |
448552 ( 1.015 ) |
136624 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
19086581.0 ( 2.3x ) |
448474 ( 1.015 ) |
136624 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17699465.0 ( 2.5x ) |
448498 ( 1.015 ) |
136624 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17175117.0 ( 2.6x ) |
448724 ( 1.016 ) |
136624 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
16959528.0 ( 2.6x ) |
448818 ( 1.016 ) |
136624 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
16957090.0 ( 2.6x ) |
460950 ( 1.043 ) |
136624 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Original data
Click here to download the raw files for this benchmark.