Benchmarks 2024 11 26 TFLM GCC Os spike_rv64 - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
eb0a3e2b0a7c57522928be39de95cd9f8c6dc636 - Spike PK :
fix-gcc14-rvv
- Spike :
Toolchains
- RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Vector:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TFLM :
8eb6b23de4470d6a8da3131650d6a67514dfa130
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: gcc, Flags: -Os, Target: spike_rv64 )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
155383021.0 ( 0.1x ) |
130994 ( 0.866 ) |
38384 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
155383040.0 ( 0.1x ) |
131314 ( 0.868 ) |
38416 ( 1.001 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
155383040.0 ( 0.1x ) |
131300 ( 0.868 ) |
38416 ( 1.001 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
155383040.0 ( 0.1x ) |
131300 ( 0.868 ) |
38416 ( 1.001 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
155383040.0 ( 0.1x ) |
131300 ( 0.868 ) |
38416 ( 1.001 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
155383040.0 ( 0.1x ) |
131300 ( 0.868 ) |
38416 ( 1.001 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
155383040.0 ( 0.1x ) |
131300 ( 0.868 ) |
38416 ( 1.001 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
17212142.0 ( Base ) |
151260 ( Base ) |
38392 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
17484459.0 ( 1.0x ) |
150964 ( 0.998 ) |
38392 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
17212162.0 ( 1.0x ) |
151718 ( 1.003 ) |
38424 ( 1.001 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17212162.0 ( 1.0x ) |
151704 ( 1.003 ) |
38424 ( 1.001 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17212162.0 ( 1.0x ) |
151704 ( 1.003 ) |
38424 ( 1.001 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17212162.0 ( 1.0x ) |
151704 ( 1.003 ) |
38424 ( 1.001 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17212162.0 ( 1.0x ) |
151704 ( 1.003 ) |
38424 ( 1.001 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17212162.0 ( 1.0x ) |
151704 ( 1.003 ) |
38424 ( 1.001 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
7107395.0 ( 2.4x ) |
154410 ( 1.021 ) |
38392 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
4854221.0 ( 3.5x ) |
154410 ( 1.021 ) |
38392 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
3618303.0 ( 4.8x ) |
154410 ( 1.021 ) |
38392 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
3577712.0 ( 4.8x ) |
154410 ( 1.021 ) |
38392 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
3577712.0 ( 4.8x ) |
154410 ( 1.021 ) |
38392 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
3579577.0 ( 4.8x ) |
154410 ( 1.021 ) |
38392 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
17484480.0 ( 1.0x ) |
151422 ( 1.001 ) |
38424 ( 1.001 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17484480.0 ( 1.0x ) |
151408 ( 1.001 ) |
38424 ( 1.001 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17484480.0 ( 1.0x ) |
151408 ( 1.001 ) |
38424 ( 1.001 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17484480.0 ( 1.0x ) |
151408 ( 1.001 ) |
38424 ( 1.001 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17484480.0 ( 1.0x ) |
151408 ( 1.001 ) |
38424 ( 1.001 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
17484480.0 ( 1.0x ) |
151408 ( 1.001 ) |
38424 ( 1.001 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
650569643.0 ( 0.1x ) |
171624 ( 0.929 ) |
71152 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
650564908.0 ( 0.1x ) |
171880 ( 0.93 ) |
71184 ( 1.001 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
650564914.0 ( 0.1x ) |
171888 ( 0.93 ) |
71184 ( 1.001 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
650564914.0 ( 0.1x ) |
171888 ( 0.93 ) |
71184 ( 1.001 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
650564914.0 ( 0.1x ) |
171888 ( 0.93 ) |
71184 ( 1.001 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
650564914.0 ( 0.1x ) |
171888 ( 0.93 ) |
71184 ( 1.001 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
650564914.0 ( 0.1x ) |
171888 ( 0.93 ) |
71184 ( 1.001 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
83908221.0 ( Base ) |
184838 ( Base ) |
71136 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
78077440.0 ( 1.1x ) |
184560 ( 0.998 ) |
71136 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
83921555.0 ( 1.0x ) |
185452 ( 1.003 ) |
71168 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
83913239.0 ( 1.0x ) |
185242 ( 1.002 ) |
71176 ( 1.001 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
83913239.0 ( 1.0x ) |
185242 ( 1.002 ) |
71176 ( 1.001 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
83913239.0 ( 1.0x ) |
185242 ( 1.002 ) |
71176 ( 1.001 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
83913239.0 ( 1.0x ) |
185242 ( 1.002 ) |
71176 ( 1.001 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
83913239.0 ( 1.0x ) |
185242 ( 1.002 ) |
71176 ( 1.001 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
28299062.0 ( 3.0x ) |
189232 ( 1.024 ) |
71136 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
17518778.0 ( 4.8x ) |
189232 ( 1.024 ) |
71136 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
12603772.0 ( 6.7x ) |
189232 ( 1.024 ) |
71136 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
10207669.0 ( 8.2x ) |
189232 ( 1.024 ) |
71136 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
8414765.0 ( 10.0x ) |
189232 ( 1.024 ) |
71136 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
7935922.0 ( 10.6x ) |
189232 ( 1.024 ) |
71136 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
78076874.0 ( 1.1x ) |
185174 ( 1.002 ) |
71168 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
78081748.0 ( 1.1x ) |
184964 ( 1.001 ) |
71176 ( 1.001 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
78081748.0 ( 1.1x ) |
184964 ( 1.001 ) |
71176 ( 1.001 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
78081748.0 ( 1.1x ) |
184964 ( 1.001 ) |
71176 ( 1.001 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
78081748.0 ( 1.1x ) |
184964 ( 1.001 ) |
71176 ( 1.001 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
78081748.0 ( 1.1x ) |
184964 ( 1.001 ) |
71176 ( 1.001 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
3863842.0 ( 0.5x ) |
332130 ( 0.986 ) |
21512 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
3863851.0 ( 0.5x ) |
332310 ( 0.986 ) |
21512 ( 1.0 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
3863851.0 ( 0.5x ) |
332296 ( 0.986 ) |
21512 ( 1.0 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
3863851.0 ( 0.5x ) |
332296 ( 0.986 ) |
21512 ( 1.0 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
3863851.0 ( 0.5x ) |
332296 ( 0.986 ) |
21512 ( 1.0 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
3863851.0 ( 0.5x ) |
332296 ( 0.986 ) |
21512 ( 1.0 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
3863851.0 ( 0.5x ) |
332296 ( 0.986 ) |
21512 ( 1.0 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
1958295.0 ( Base ) |
336914 ( Base ) |
21512 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
3574779.0 ( 0.5x ) |
336918 ( 1.0 ) |
21512 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
1944951.0 ( 1.0x ) |
337228 ( 1.001 ) |
21512 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
1944951.0 ( 1.0x ) |
337214 ( 1.001 ) |
21512 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
1944951.0 ( 1.0x ) |
337214 ( 1.001 ) |
21512 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
1944951.0 ( 1.0x ) |
337214 ( 1.001 ) |
21512 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
1944951.0 ( 1.0x ) |
337214 ( 1.001 ) |
21512 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
1944951.0 ( 1.0x ) |
337214 ( 1.001 ) |
21512 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
2466306.0 ( 0.8x ) |
340108 ( 1.009 ) |
21512 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2231978.0 ( 0.9x ) |
340108 ( 1.009 ) |
21512 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2114814.0 ( 0.9x ) |
340108 ( 1.009 ) |
21512 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2056772.0 ( 1.0x ) |
340108 ( 1.009 ) |
21512 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2051352.0 ( 1.0x ) |
340108 ( 1.009 ) |
21512 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
2047642.0 ( 1.0x ) |
340108 ( 1.009 ) |
21512 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
3561394.0 ( 0.5x ) |
337232 ( 1.001 ) |
21512 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
3561371.0 ( 0.5x ) |
337218 ( 1.001 ) |
21512 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
3561371.0 ( 0.5x ) |
337218 ( 1.001 ) |
21512 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
3561371.0 ( 0.5x ) |
337218 ( 1.001 ) |
21512 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
3561371.0 ( 0.5x ) |
337218 ( 1.001 ) |
21512 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
3561371.0 ( 0.5x ) |
337218 ( 1.001 ) |
21512 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
438462070.0 ( 0.1x ) |
404636 ( 0.952 ) |
136688 ( 1.0 ) |
0 | TFLM | Reference | RV64GC | 0 | - |
438462212.0 ( 0.1x ) |
404956 ( 0.953 ) |
136720 ( 1.0 ) |
128 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
438462212.0 ( 0.1x ) |
404942 ( 0.953 ) |
136720 ( 1.0 ) |
256 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
438462212.0 ( 0.1x ) |
404942 ( 0.953 ) |
136720 ( 1.0 ) |
512 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
438462212.0 ( 0.1x ) |
404942 ( 0.953 ) |
136720 ( 1.0 ) |
1024 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
438462212.0 ( 0.1x ) |
404942 ( 0.953 ) |
136720 ( 1.0 ) |
2048 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
438462212.0 ( 0.1x ) |
404942 ( 0.953 ) |
136720 ( 1.0 ) |
4096 | TFLM | Reference | RV64GCV | 0 | Loop+SLP |
51260363.0 ( Base ) |
424902 ( Base ) |
136696 ( Base ) |
0 | muRISCV-NN | Scalar | RV64GC | 0 | - |
52975759.0 ( 1.0x ) |
424606 ( 0.999 ) |
136696 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV64GC | 0 | - |
51260348.0 ( 1.0x ) |
425368 ( 1.001 ) |
136728 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
51260448.0 ( 1.0x ) |
425354 ( 1.001 ) |
136728 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
51260448.0 ( 1.0x ) |
425354 ( 1.001 ) |
136728 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
51260448.0 ( 1.0x ) |
425354 ( 1.001 ) |
136728 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
51260448.0 ( 1.0x ) |
425354 ( 1.001 ) |
136728 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
51260448.0 ( 1.0x ) |
425354 ( 1.001 ) |
136728 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
22898336.0 ( 2.2x ) |
428052 ( 1.007 ) |
136696 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV64GCV | 0 | - |
16945676.0 ( 3.0x ) |
428052 ( 1.007 ) |
136696 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV64GCV | 0 | - |
14661770.0 ( 3.5x ) |
428052 ( 1.007 ) |
136696 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV64GCV | 0 | - |
13757196.0 ( 3.7x ) |
428052 ( 1.007 ) |
136696 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV64GCV | 0 | - |
13680116.0 ( 3.7x ) |
428052 ( 1.007 ) |
136696 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV64GCV | 0 | - |
13681981.0 ( 3.7x ) |
428052 ( 1.007 ) |
136696 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV64GCV | 0 | - |
52970661.0 ( 1.0x ) |
425072 ( 1.0 ) |
136728 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
52970661.0 ( 1.0x ) |
425058 ( 1.0 ) |
136728 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
52970661.0 ( 1.0x ) |
425058 ( 1.0 ) |
136728 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
52970661.0 ( 1.0x ) |
425058 ( 1.0 ) |
136728 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
52970661.0 ( 1.0x ) |
425058 ( 1.0 ) |
136728 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
52970661.0 ( 1.0x ) |
425058 ( 1.0 ) |
136728 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV64GCV | 0 | Loop+SLP |
Original data
Click here to download the raw files for this benchmark.