Benchmarks 2024 11 26 TFLM GCC Os spike_rv32 - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
eb0a3e2b0a7c57522928be39de95cd9f8c6dc636 - Spike PK :
fix-gcc14-rvv
- Spike :
Toolchains
- RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Vector:
riscv32-unknown-elf-gcc (g8b4bb54e6c4) 14.2.1 20241118 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TFLM :
8eb6b23de4470d6a8da3131650d6a67514dfa130
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: gcc, Flags: -Os, Target: spike_rv32 )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
175176332.0 ( 0.1x ) |
134982 ( 0.878 ) |
36140 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
175171178.0 ( 0.1x ) |
135290 ( 0.88 ) |
36172 ( 1.001 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
175171178.0 ( 0.1x ) |
135290 ( 0.88 ) |
36172 ( 1.001 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
175171178.0 ( 0.1x ) |
135290 ( 0.88 ) |
36172 ( 1.001 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
175171178.0 ( 0.1x ) |
135290 ( 0.88 ) |
36172 ( 1.001 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
175171178.0 ( 0.1x ) |
135290 ( 0.88 ) |
36172 ( 1.001 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
175171178.0 ( 0.1x ) |
135290 ( 0.88 ) |
36172 ( 1.001 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
17307266.0 ( Base ) |
153668 ( Base ) |
36148 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
17313741.0 ( 1.0x ) |
153548 ( 0.999 ) |
36148 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
17306778.0 ( 1.0x ) |
154030 ( 1.002 ) |
36180 ( 1.001 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17306778.0 ( 1.0x ) |
154030 ( 1.002 ) |
36180 ( 1.001 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17306778.0 ( 1.0x ) |
154030 ( 1.002 ) |
36180 ( 1.001 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17306778.0 ( 1.0x ) |
154030 ( 1.002 ) |
36180 ( 1.001 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17306778.0 ( 1.0x ) |
154030 ( 1.002 ) |
36180 ( 1.001 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17306778.0 ( 1.0x ) |
154030 ( 1.002 ) |
36180 ( 1.001 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
4636236.0 ( 3.7x ) |
156278 ( 1.017 ) |
36148 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
3296310.0 ( 5.3x ) |
156278 ( 1.017 ) |
36148 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2567596.0 ( 6.7x ) |
156278 ( 1.017 ) |
36148 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2525716.0 ( 6.9x ) |
156278 ( 1.017 ) |
36148 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2529107.0 ( 6.8x ) |
156278 ( 1.017 ) |
36148 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2532497.0 ( 6.8x ) |
156278 ( 1.017 ) |
36148 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
17313754.0 ( 1.0x ) |
153908 ( 1.002 ) |
36180 ( 1.001 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17313754.0 ( 1.0x ) |
153908 ( 1.002 ) |
36180 ( 1.001 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17313754.0 ( 1.0x ) |
153908 ( 1.002 ) |
36180 ( 1.001 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17313754.0 ( 1.0x ) |
153908 ( 1.002 ) |
36180 ( 1.001 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17313754.0 ( 1.0x ) |
153908 ( 1.002 ) |
36180 ( 1.001 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
17313754.0 ( 1.0x ) |
153908 ( 1.002 ) |
36180 ( 1.001 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
746696943.0 ( 0.1x ) |
175586 ( 0.934 ) |
68904 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
746696807.0 ( 0.1x ) |
175834 ( 0.935 ) |
68936 ( 1.001 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
746696636.0 ( 0.1x ) |
175860 ( 0.935 ) |
68940 ( 1.001 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
746696636.0 ( 0.1x ) |
175860 ( 0.935 ) |
68940 ( 1.001 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
746696636.0 ( 0.1x ) |
175860 ( 0.935 ) |
68940 ( 1.001 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
746696636.0 ( 0.1x ) |
175860 ( 0.935 ) |
68940 ( 1.001 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
746696636.0 ( 0.1x ) |
175860 ( 0.935 ) |
68940 ( 1.001 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
79928051.0 ( Base ) |
188050 ( Base ) |
68896 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
79019111.0 ( 1.0x ) |
187804 ( 0.999 ) |
68896 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
79928264.0 ( 1.0x ) |
188566 ( 1.003 ) |
68928 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
79928042.0 ( 1.0x ) |
188370 ( 1.002 ) |
68932 ( 1.001 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
79928042.0 ( 1.0x ) |
188370 ( 1.002 ) |
68932 ( 1.001 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
79928042.0 ( 1.0x ) |
188370 ( 1.002 ) |
68932 ( 1.001 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
79928042.0 ( 1.0x ) |
188370 ( 1.002 ) |
68932 ( 1.001 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
79928042.0 ( 1.0x ) |
188370 ( 1.002 ) |
68932 ( 1.001 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
17065996.0 ( 4.7x ) |
191494 ( 1.018 ) |
68896 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
11009880.0 ( 7.3x ) |
191494 ( 1.018 ) |
68896 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8248062.0 ( 9.7x ) |
191494 ( 1.018 ) |
68896 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
6900510.0 ( 11.6x ) |
191494 ( 1.018 ) |
68896 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
5897325.0 ( 13.6x ) |
191494 ( 1.018 ) |
68896 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
5630387.0 ( 14.2x ) |
191494 ( 1.018 ) |
68896 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
79024517.0 ( 1.0x ) |
188320 ( 1.001 ) |
68928 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
79024240.0 ( 1.0x ) |
188124 ( 1.0 ) |
68932 ( 1.001 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
79024240.0 ( 1.0x ) |
188124 ( 1.0 ) |
68932 ( 1.001 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
79024240.0 ( 1.0x ) |
188124 ( 1.0 ) |
68932 ( 1.001 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
79024240.0 ( 1.0x ) |
188124 ( 1.0 ) |
68932 ( 1.001 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
79024240.0 ( 1.0x ) |
188124 ( 1.0 ) |
68932 ( 1.001 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
3119139.0 ( 0.6x ) |
336328 ( 0.987 ) |
19424 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
3119091.0 ( 0.6x ) |
336484 ( 0.987 ) |
19424 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3119091.0 ( 0.6x ) |
336484 ( 0.987 ) |
19424 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3119091.0 ( 0.6x ) |
336484 ( 0.987 ) |
19424 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3119091.0 ( 0.6x ) |
336484 ( 0.987 ) |
19424 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3119091.0 ( 0.6x ) |
336484 ( 0.987 ) |
19424 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3119091.0 ( 0.6x ) |
336484 ( 0.987 ) |
19424 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
1839069.0 ( Base ) |
340768 ( Base ) |
19424 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
3188381.0 ( 0.6x ) |
340770 ( 1.0 ) |
19424 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
1839140.0 ( 1.0x ) |
341054 ( 1.001 ) |
19424 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1839140.0 ( 1.0x ) |
341054 ( 1.001 ) |
19424 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1839140.0 ( 1.0x ) |
341054 ( 1.001 ) |
19424 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1839140.0 ( 1.0x ) |
341054 ( 1.001 ) |
19424 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1839140.0 ( 1.0x ) |
341054 ( 1.001 ) |
19424 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1839140.0 ( 1.0x ) |
341054 ( 1.001 ) |
19424 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1981796.0 ( 0.9x ) |
343088 ( 1.007 ) |
19424 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1854988.0 ( 1.0x ) |
343088 ( 1.007 ) |
19424 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1791584.0 ( 1.0x ) |
343088 ( 1.007 ) |
19424 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1760278.0 ( 1.0x ) |
343088 ( 1.007 ) |
19424 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1756329.0 ( 1.0x ) |
343088 ( 1.007 ) |
19424 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1757696.0 ( 1.0x ) |
343088 ( 1.007 ) |
19424 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
3188484.0 ( 0.6x ) |
341056 ( 1.001 ) |
19424 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
3188484.0 ( 0.6x ) |
341056 ( 1.001 ) |
19424 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
3188484.0 ( 0.6x ) |
341056 ( 1.001 ) |
19424 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
3188484.0 ( 0.6x ) |
341056 ( 1.001 ) |
19424 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
3188484.0 ( 0.6x ) |
341056 ( 1.001 ) |
19424 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
3188484.0 ( 0.6x ) |
341056 ( 1.001 ) |
19424 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
496732695.0 ( 0.1x ) |
408624 ( 0.956 ) |
134444 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
496728033.0 ( 0.1x ) |
408932 ( 0.957 ) |
134476 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
496728033.0 ( 0.1x ) |
408932 ( 0.957 ) |
134476 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
496728033.0 ( 0.1x ) |
408932 ( 0.957 ) |
134476 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
496728033.0 ( 0.1x ) |
408932 ( 0.957 ) |
134476 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
496728033.0 ( 0.1x ) |
408932 ( 0.957 ) |
134476 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
496728033.0 ( 0.1x ) |
408932 ( 0.957 ) |
134476 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
51813880.0 ( Base ) |
427310 ( Base ) |
134452 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
52777327.0 ( 1.0x ) |
427190 ( 1.0 ) |
134452 ( 1.0 ) |
0 | muRISCV-NN | Vector (Portable) | RV32GC | 0 | - |
51809659.0 ( 1.0x ) |
427680 ( 1.001 ) |
134484 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
51809659.0 ( 1.0x ) |
427680 ( 1.001 ) |
134484 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
51809659.0 ( 1.0x ) |
427680 ( 1.001 ) |
134484 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
51809659.0 ( 1.0x ) |
427680 ( 1.001 ) |
134484 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
51809659.0 ( 1.0x ) |
427680 ( 1.001 ) |
134484 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
51809659.0 ( 1.0x ) |
427680 ( 1.001 ) |
134484 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
15302391.0 ( 3.4x ) |
429920 ( 1.006 ) |
134452 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
11783439.0 ( 4.4x ) |
429920 ( 1.006 ) |
134452 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
10422267.0 ( 5.0x ) |
429920 ( 1.006 ) |
134452 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
9888617.0 ( 5.2x ) |
429920 ( 1.006 ) |
134452 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
9841262.0 ( 5.3x ) |
429920 ( 1.006 ) |
134452 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
9844652.0 ( 5.3x ) |
429920 ( 1.006 ) |
134452 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
52784294.0 ( 1.0x ) |
427558 ( 1.001 ) |
134484 ( 1.0 ) |
128 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
52784294.0 ( 1.0x ) |
427558 ( 1.001 ) |
134484 ( 1.0 ) |
256 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
52784294.0 ( 1.0x ) |
427558 ( 1.001 ) |
134484 ( 1.0 ) |
512 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
52784294.0 ( 1.0x ) |
427558 ( 1.001 ) |
134484 ( 1.0 ) |
1024 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
52784294.0 ( 1.0x ) |
427558 ( 1.001 ) |
134484 ( 1.0 ) |
2048 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
52784294.0 ( 1.0x ) |
427558 ( 1.001 ) |
134484 ( 1.0 ) |
4096 | muRISCV-NN | Vector (Portable) | RV32GCV | 0 | Loop+SLP |
Original data
Click here to download the raw files for this benchmark.