Benchmarks 2024 11 18 TFLM GCC Os spike_rv32 - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
0bc176b3fca43560b9e8586cdbc41cfde073e17a - Spike PK :
7e9b671c0415dfd7b562ac934feb9380075d4aa2
- Spike :
Toolchains
- RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (gc891d8dc23e) 13.2.0 - Vector:
riscv32-unknown-elf-gcc (gc891d8dc23e) 13.2.0 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TFLM :
8eb6b23de4470d6a8da3131650d6a67514dfa130
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: gcc, Flags: -Os, Target: spike_rv32 )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
174712306 ( 0.1x ) |
132180 ( 0.876 ) |
36140 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
174712294 ( 0.1x ) |
132186 ( 0.876 ) |
36140 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
174712294 ( 0.1x ) |
132186 ( 0.876 ) |
36140 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
174712294 ( 0.1x ) |
132186 ( 0.876 ) |
36140 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
174712294 ( 0.1x ) |
132186 ( 0.876 ) |
36140 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
174712294 ( 0.1x ) |
132186 ( 0.876 ) |
36140 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
174712294 ( 0.1x ) |
132186 ( 0.876 ) |
36140 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
16660068 ( Base ) |
150858 ( Base ) |
36148 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
16660068 ( 1.0x ) |
150858 ( 1.0 ) |
36148 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
16660068 ( 1.0x ) |
150858 ( 1.0 ) |
36148 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
16660068 ( 1.0x ) |
150858 ( 1.0 ) |
36148 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
16660068 ( 1.0x ) |
150858 ( 1.0 ) |
36148 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
16660068 ( 1.0x ) |
150858 ( 1.0 ) |
36148 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
16660068 ( 1.0x ) |
150858 ( 1.0 ) |
36148 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
4117424 ( 4.0x ) |
152574 ( 1.011 ) |
36148 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2849368 ( 5.8x ) |
152574 ( 1.011 ) |
36148 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2160680 ( 7.7x ) |
152574 ( 1.011 ) |
36148 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2118968 ( 7.9x ) |
152574 ( 1.011 ) |
36148 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2118968 ( 7.9x ) |
152574 ( 1.011 ) |
36148 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
2122357 ( 7.8x ) |
152574 ( 1.011 ) |
36148 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
745812529 ( 0.1x ) |
172688 ( 0.932 ) |
68904 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
745812529 ( 0.1x ) |
172704 ( 0.932 ) |
68904 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
745812529 ( 0.1x ) |
172704 ( 0.932 ) |
68904 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
745812529 ( 0.1x ) |
172704 ( 0.932 ) |
68904 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
745812529 ( 0.1x ) |
172704 ( 0.932 ) |
68904 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
745812529 ( 0.1x ) |
172704 ( 0.932 ) |
68904 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
745812529 ( 0.1x ) |
172704 ( 0.932 ) |
68904 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
81003528 ( Base ) |
185254 ( Base ) |
68896 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
81003528 ( 1.0x ) |
185280 ( 1.0 ) |
68896 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
81003528 ( 1.0x ) |
185280 ( 1.0 ) |
68896 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
81003528 ( 1.0x ) |
185280 ( 1.0 ) |
68896 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
81003528 ( 1.0x ) |
185280 ( 1.0 ) |
68896 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
81003528 ( 1.0x ) |
185280 ( 1.0 ) |
68896 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
81003528 ( 1.0x ) |
185280 ( 1.0 ) |
68896 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
15480213 ( 5.2x ) |
187806 ( 1.014 ) |
68896 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
9793205 ( 8.3x ) |
187806 ( 1.014 ) |
68896 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
7199557 ( 11.3x ) |
187806 ( 1.014 ) |
68896 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
5934037 ( 13.7x ) |
187806 ( 1.014 ) |
68896 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
4992346 ( 16.2x ) |
187806 ( 1.014 ) |
68896 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
4741819 ( 17.1x ) |
187806 ( 1.014 ) |
68896 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
3106421 ( 0.6x ) |
333552 ( 0.987 ) |
19424 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
3106421 ( 0.6x ) |
333558 ( 0.987 ) |
19424 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3106421 ( 0.6x ) |
333558 ( 0.987 ) |
19424 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3106421 ( 0.6x ) |
333558 ( 0.987 ) |
19424 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3106421 ( 0.6x ) |
333558 ( 0.987 ) |
19424 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3106421 ( 0.6x ) |
333558 ( 0.987 ) |
19424 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
3106421 ( 0.6x ) |
333558 ( 0.987 ) |
19424 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
1777790 ( Base ) |
337930 ( Base ) |
19424 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
1777790 ( 1.0x ) |
337930 ( 1.0 ) |
19424 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1777790 ( 1.0x ) |
337930 ( 1.0 ) |
19424 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1777790 ( 1.0x ) |
337930 ( 1.0 ) |
19424 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1777790 ( 1.0x ) |
337930 ( 1.0 ) |
19424 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1777790 ( 1.0x ) |
337930 ( 1.0 ) |
19424 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1777790 ( 1.0x ) |
337930 ( 1.0 ) |
19424 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
1942762 ( 0.9x ) |
339476 ( 1.005 ) |
19424 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1824394 ( 1.0x ) |
339476 ( 1.005 ) |
19424 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1765210 ( 1.0x ) |
339476 ( 1.005 ) |
19424 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1735954 ( 1.0x ) |
339476 ( 1.005 ) |
19424 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1732270 ( 1.0x ) |
339476 ( 1.005 ) |
19424 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
1730386 ( 1.0x ) |
339476 ( 1.005 ) |
19424 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|
495272719 ( 0.1x ) |
405822 ( 0.956 ) |
134444 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | 0 | - |
495272719 ( 0.1x ) |
405828 ( 0.956 ) |
134444 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
495272719 ( 0.1x ) |
405828 ( 0.956 ) |
134444 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
495272719 ( 0.1x ) |
405828 ( 0.956 ) |
134444 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
495272719 ( 0.1x ) |
405828 ( 0.956 ) |
134444 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
495272719 ( 0.1x ) |
405828 ( 0.956 ) |
134444 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
495272719 ( 0.1x ) |
405828 ( 0.956 ) |
134444 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | 0 | Loop+SLP |
49690025 ( Base ) |
424500 ( Base ) |
134452 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | 0 | - |
49690025 ( 1.0x ) |
424500 ( 1.0 ) |
134452 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
49690025 ( 1.0x ) |
424500 ( 1.0 ) |
134452 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
49690025 ( 1.0x ) |
424500 ( 1.0 ) |
134452 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
49690025 ( 1.0x ) |
424500 ( 1.0 ) |
134452 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
49690025 ( 1.0x ) |
424500 ( 1.0 ) |
134452 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
49690025 ( 1.0x ) |
424500 ( 1.0 ) |
134452 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | 0 | Loop+SLP |
13489800 ( 3.7x ) |
426216 ( 1.004 ) |
134452 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | 0 | - |
10161172 ( 4.9x ) |
426216 ( 1.004 ) |
134452 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8872050 ( 5.6x ) |
426216 ( 1.004 ) |
134452 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8367551 ( 5.9x ) |
426216 ( 1.004 ) |
134452 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8318937 ( 6.0x ) |
426216 ( 1.004 ) |
134452 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | 0 | - |
8322326 ( 6.0x ) |
426216 ( 1.004 ) |
134452 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | 0 | - |
Original data
Click here to download the raw files for this benchmark.