Benchmarks 2024 11 26 TFLM LLVM O3 spike_rv32 - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
eb0a3e2b0a7c57522928be39de95cd9f8c6dc636 - Spike PK :
fix-gcc14-rvv
- Spike :
Toolchains
-
RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Vector:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
-
LLVM/Clang:
clang version 18.1.8 (https://github.com/llvm/llvm-project.git 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)- Linker: lld (TODO)
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TFLM :
8eb6b23de4470d6a8da3131650d6a67514dfa130
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: llvm, Flags: -O3, Target: spike_rv32 )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
38583516.0 ( 0.4x ) |
159832 ( 0.893 ) |
36084 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
28766247.0 ( 0.5x ) |
175122 ( 0.978 ) |
36092 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
27360974.0 ( 0.6x ) |
175378 ( 0.98 ) |
36092 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
26656775.0 ( 0.6x ) |
175724 ( 0.981 ) |
36092 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
26302901.0 ( 0.6x ) |
175982 ( 0.983 ) |
36092 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
42149645.0 ( 0.4x ) |
176202 ( 0.984 ) |
36092 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
42156427.0 ( 0.4x ) |
177162 ( 0.989 ) |
36092 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
15103433.0 ( Base ) |
179046 ( Base ) |
36084 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
14903933.0 ( 1.0x ) |
178834 ( 0.999 ) |
36084 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
5347285.0 ( 2.8x ) |
197814 ( 1.105 ) |
36092 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
4930949.0 ( 3.1x ) |
198138 ( 1.107 ) |
36092 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
4612940.0 ( 3.3x ) |
198392 ( 1.108 ) |
36092 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
5360735.0 ( 2.8x ) |
198540 ( 1.109 ) |
36092 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
13835200.0 ( 1.1x ) |
198650 ( 1.109 ) |
36092 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
13828301.0 ( 1.1x ) |
199218 ( 1.113 ) |
36092 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
4064607.0 ( 3.7x ) |
181424 ( 1.013 ) |
36084 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2809645.0 ( 5.4x ) |
181452 ( 1.013 ) |
36084 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2132335.0 ( 7.1x ) |
181416 ( 1.013 ) |
36084 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2091351.0 ( 7.2x ) |
181454 ( 1.013 ) |
36084 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2094742.0 ( 7.2x ) |
181454 ( 1.013 ) |
36084 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2098133.0 ( 7.2x ) |
181454 ( 1.013 ) |
36084 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
6398459.0 ( 2.4x ) |
198508 ( 1.109 ) |
36092 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
5432019.0 ( 2.8x ) |
198804 ( 1.11 ) |
36092 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
5097002.0 ( 3.0x ) |
199066 ( 1.112 ) |
36092 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
6181653.0 ( 2.4x ) |
199214 ( 1.113 ) |
36092 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
14648560.0 ( 1.0x ) |
199306 ( 1.113 ) |
36092 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
14651951.0 ( 1.0x ) |
199874 ( 1.116 ) |
36092 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
120042844.0 ( 0.5x ) |
200020 ( 0.941 ) |
68852 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
52315145.0 ( 1.1x ) |
217084 ( 1.021 ) |
68860 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
44654160.0 ( 1.3x ) |
217438 ( 1.023 ) |
68860 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
82961728.0 ( 0.7x ) |
218012 ( 1.026 ) |
68860 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
107873026.0 ( 0.5x ) |
218392 ( 1.027 ) |
68860 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
123206621.0 ( 0.5x ) |
218732 ( 1.029 ) |
68860 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
123223693.0 ( 0.5x ) |
220132 ( 1.036 ) |
68860 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
56212127.0 ( Base ) |
212558 ( Base ) |
68848 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
72277759.0 ( 0.8x ) |
212186 ( 0.998 ) |
68848 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
11808236.0 ( 4.8x ) |
234276 ( 1.102 ) |
68856 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
11356182.0 ( 4.9x ) |
234614 ( 1.104 ) |
68856 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
14221400.0 ( 4.0x ) |
235128 ( 1.106 ) |
68856 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
15712058.0 ( 3.6x ) |
235418 ( 1.108 ) |
68856 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
16394497.0 ( 3.4x ) |
235672 ( 1.109 ) |
68856 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
35580201.0 ( 1.6x ) |
236666 ( 1.113 ) |
68856 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
15153355.0 ( 3.7x ) |
215228 ( 1.013 ) |
68848 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
9570159.0 ( 5.9x ) |
215248 ( 1.013 ) |
68848 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
7024321.0 ( 8.0x ) |
215212 ( 1.012 ) |
68848 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
5782193.0 ( 9.7x ) |
215250 ( 1.013 ) |
68848 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
4857320.0 ( 11.6x ) |
215250 ( 1.013 ) |
68848 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
4611619.0 ( 12.2x ) |
215250 ( 1.013 ) |
68848 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
17594933.0 ( 3.2x ) |
233742 ( 1.1 ) |
68856 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
16562252.0 ( 3.4x ) |
234080 ( 1.101 ) |
68856 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17328032.0 ( 3.2x ) |
234598 ( 1.104 ) |
68856 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17041373.0 ( 3.3x ) |
234888 ( 1.105 ) |
68856 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
16582090.0 ( 3.4x ) |
235142 ( 1.106 ) |
68856 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
43363339.0 ( 1.3x ) |
236136 ( 1.111 ) |
68856 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
2804439.0 ( 0.6x ) |
345800 ( 0.983 ) |
19372 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
800177.0 ( 2.1x ) |
349262 ( 0.993 ) |
19372 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
626673.0 ( 2.7x ) |
349358 ( 0.993 ) |
19372 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
531064.0 ( 3.2x ) |
349482 ( 0.994 ) |
19372 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
487504.0 ( 3.5x ) |
349614 ( 0.994 ) |
19372 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
464888.0 ( 3.6x ) |
349714 ( 0.994 ) |
19372 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
2220296.0 ( 0.8x ) |
349824 ( 0.995 ) |
19372 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
1689644.0 ( Base ) |
351696 ( Base ) |
19372 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
3038990.0 ( 0.6x ) |
351698 ( 1.0 ) |
19372 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
552790.0 ( 3.1x ) |
356752 ( 1.014 ) |
19372 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
465571.0 ( 3.6x ) |
356830 ( 1.015 ) |
19372 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
420360.0 ( 4.0x ) |
356904 ( 1.015 ) |
19372 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
402900.0 ( 4.2x ) |
356998 ( 1.015 ) |
19372 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
391588.0 ( 4.3x ) |
357060 ( 1.015 ) |
19372 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1328532.0 ( 1.3x ) |
357124 ( 1.015 ) |
19372 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1937850.0 ( 0.9x ) |
352802 ( 1.003 ) |
19372 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1821170.0 ( 0.9x ) |
352788 ( 1.003 ) |
19372 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1762829.0 ( 1.0x ) |
352756 ( 1.003 ) |
19372 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1733984.0 ( 1.0x ) |
352794 ( 1.003 ) |
19372 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1730353.0 ( 1.0x ) |
352794 ( 1.003 ) |
19372 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1728497.0 ( 1.0x ) |
352794 ( 1.003 ) |
19372 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
789422.0 ( 2.1x ) |
356754 ( 1.014 ) |
19372 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
607480.0 ( 2.8x ) |
356832 ( 1.015 ) |
19372 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
512931.0 ( 3.3x ) |
356906 ( 1.015 ) |
19372 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
470794.0 ( 3.6x ) |
357000 ( 1.015 ) |
19372 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
447157.0 ( 3.8x ) |
357062 ( 1.015 ) |
19372 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
2357596.0 ( 0.7x ) |
357126 ( 1.015 ) |
19372 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
101694057.0 ( 0.4x ) |
433474 ( 0.958 ) |
134388 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
69591099.0 ( 0.6x ) |
448764 ( 0.991 ) |
134396 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
67271682.0 ( 0.7x ) |
449020 ( 0.992 ) |
134396 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
67391392.0 ( 0.7x ) |
449366 ( 0.993 ) |
134396 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
73186430.0 ( 0.6x ) |
449624 ( 0.993 ) |
134396 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
79704928.0 ( 0.6x ) |
449844 ( 0.994 ) |
134396 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
105202232.0 ( 0.4x ) |
450572 ( 0.995 ) |
134396 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
45215610.0 ( Base ) |
452688 ( Base ) |
134388 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
44844965.0 ( 1.0x ) |
452476 ( 1.0 ) |
134388 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
17462016.0 ( 2.6x ) |
471456 ( 1.041 ) |
134396 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17039215.0 ( 2.7x ) |
471780 ( 1.042 ) |
134396 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
18215342.0 ( 2.5x ) |
472034 ( 1.043 ) |
134396 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
21361663.0 ( 2.1x ) |
472182 ( 1.043 ) |
134396 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
24888125.0 ( 1.8x ) |
472292 ( 1.043 ) |
134396 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
38847252.0 ( 1.2x ) |
473140 ( 1.045 ) |
134396 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
13253929.0 ( 3.4x ) |
455066 ( 1.005 ) |
134388 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
9932959.0 ( 4.6x ) |
455094 ( 1.005 ) |
134388 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8641538.0 ( 5.2x ) |
455058 ( 1.005 ) |
134388 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8137559.0 ( 5.6x ) |
455096 ( 1.005 ) |
134388 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8093184.0 ( 5.6x ) |
455096 ( 1.005 ) |
134388 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8096575.0 ( 5.6x ) |
455096 ( 1.005 ) |
134388 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
20281829.0 ( 2.2x ) |
472150 ( 1.043 ) |
134396 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
19155334.0 ( 2.4x ) |
472446 ( 1.044 ) |
134396 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
20670053.0 ( 2.2x ) |
472708 ( 1.044 ) |
134396 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
23816347.0 ( 1.9x ) |
472856 ( 1.045 ) |
134396 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
27334446.0 ( 1.7x ) |
472948 ( 1.045 ) |
134396 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
41293548.0 ( 1.1x ) |
473796 ( 1.047 ) |
134396 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Original data
Click here to download the raw files for this benchmark.