Benchmarks 2024 02 26 TFLM GCC - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)
Toolchains
- RISC-V GCC:
- Scalar: TODO: version & url
- Vector: TODO: version & url
- Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Package Versions
-
MLonMCU : main
-
TFLM : main
-
Spike : 0bc176b3fca43560b9e8586cdbc41cfde073e17a
-
Spike PK : 7e9b671c0415dfd7b562ac934feb9380075d4aa2
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tflm, Backend: tflmi, Toolchain: gcc)
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Auto-Vectorization |
|---|---|---|---|---|---|---|---|
174698646 ( 0.1x ) |
132407 ( 0.881 ) |
36204 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | - |
174698646 ( 0.1x ) |
132413 ( 0.881 ) |
36204 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | Loop+SLP |
174698646 ( 0.1x ) |
132413 ( 0.881 ) |
36204 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | Loop+SLP |
174698646 ( 0.1x ) |
132413 ( 0.881 ) |
36204 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | Loop+SLP |
174698646 ( 0.1x ) |
132413 ( 0.881 ) |
36204 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | Loop+SLP |
174698646 ( 0.1x ) |
132413 ( 0.881 ) |
36204 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | Loop+SLP |
174698646 ( 0.1x ) |
132413 ( 0.881 ) |
36204 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | Loop+SLP |
157549999 ( 0.1x ) |
144774 ( 0.963 ) |
36148 ( 0.998 ) |
0 | TFLM | Reference | RV32GCP | - |
16643398 ( Base ) |
150310 ( Base ) |
36212 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | - |
16643398 ( 1.0x ) |
150312 ( 1.0 ) |
36212 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
16643398 ( 1.0x ) |
150312 ( 1.0 ) |
36212 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
16643398 ( 1.0x ) |
150312 ( 1.0 ) |
36212 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
16643398 ( 1.0x ) |
150312 ( 1.0 ) |
36212 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
16643398 ( 1.0x ) |
150312 ( 1.0 ) |
36212 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
16643398 ( 1.0x ) |
150312 ( 1.0 ) |
36212 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
4101711 ( 4.1x ) |
151494 ( 1.008 ) |
36212 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | - |
2833655 ( 5.9x ) |
151494 ( 1.008 ) |
36212 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | - |
2144967 ( 7.8x ) |
151494 ( 1.008 ) |
36212 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | - |
2103255 ( 7.9x ) |
151494 ( 1.008 ) |
36212 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | - |
2103255 ( 7.9x ) |
151494 ( 1.008 ) |
36212 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | - |
2106644 ( 7.9x ) |
151494 ( 1.008 ) |
36212 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | - |
13496938 ( 1.2x ) |
161878 ( 1.077 ) |
36156 ( 0.998 ) |
0 | muRISCV-NN | Scalar | RV32GCP | - |
15938355 ( 1.0x ) |
164150 ( 1.092 ) |
36156 ( 0.998 ) |
0 | muRISCV-NN | Packed | RV32GCP | - |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Auto-Vectorization |
|---|---|---|---|---|---|---|---|
745801113 ( 0.1x ) |
172997 ( 0.936 ) |
68968 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | - |
745801113 ( 0.1x ) |
173013 ( 0.936 ) |
68968 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | Loop+SLP |
745801113 ( 0.1x ) |
173013 ( 0.936 ) |
68968 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | Loop+SLP |
745801113 ( 0.1x ) |
173013 ( 0.936 ) |
68968 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | Loop+SLP |
745801113 ( 0.1x ) |
173013 ( 0.936 ) |
68968 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | Loop+SLP |
745801113 ( 0.1x ) |
173013 ( 0.936 ) |
68968 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | Loop+SLP |
745801113 ( 0.1x ) |
173013 ( 0.936 ) |
68968 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | Loop+SLP |
697912970 ( 0.1x ) |
185266 ( 1.002 ) |
68912 ( 0.999 ) |
0 | TFLM | Reference | RV32GCP | - |
80991569 ( Base ) |
184814 ( Base ) |
68960 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | - |
80992072 ( 1.0x ) |
184842 ( 1.0 ) |
68960 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
80992072 ( 1.0x ) |
184842 ( 1.0 ) |
68960 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
80992072 ( 1.0x ) |
184842 ( 1.0 ) |
68960 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
80992072 ( 1.0x ) |
184842 ( 1.0 ) |
68960 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
80992072 ( 1.0x ) |
184842 ( 1.0 ) |
68960 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
80992072 ( 1.0x ) |
184842 ( 1.0 ) |
68960 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
15465187 ( 5.2x ) |
186830 ( 1.011 ) |
68960 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | - |
9778179 ( 8.3x ) |
186830 ( 1.011 ) |
68960 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | - |
7184531 ( 11.3x ) |
186830 ( 1.011 ) |
68960 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | - |
5919011 ( 13.7x ) |
186830 ( 1.011 ) |
68960 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | - |
4977320 ( 16.3x ) |
186830 ( 1.011 ) |
68960 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | - |
4726793 ( 17.1x ) |
186830 ( 1.011 ) |
68960 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | - |
62967943 ( 1.3x ) |
196482 ( 1.063 ) |
68904 ( 0.999 ) |
0 | muRISCV-NN | Scalar | RV32GCP | - |
68428908 ( 1.2x ) |
199348 ( 1.079 ) |
68904 ( 0.999 ) |
0 | muRISCV-NN | Packed | RV32GCP | - |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Auto-Vectorization |
|---|---|---|---|---|---|---|---|
3094956 ( 0.6x ) |
333908 ( 0.991 ) |
19432 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | - |
3094956 ( 0.6x ) |
333914 ( 0.991 ) |
19432 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | Loop+SLP |
3094956 ( 0.6x ) |
333914 ( 0.991 ) |
19432 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | Loop+SLP |
3094956 ( 0.6x ) |
333914 ( 0.991 ) |
19432 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | Loop+SLP |
3094956 ( 0.6x ) |
333914 ( 0.991 ) |
19432 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | Loop+SLP |
3094956 ( 0.6x ) |
333914 ( 0.991 ) |
19432 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | Loop+SLP |
3094956 ( 0.6x ) |
333914 ( 0.991 ) |
19432 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | Loop+SLP |
3097895 ( 0.6x ) |
346264 ( 1.027 ) |
19380 ( 0.997 ) |
0 | TFLM | Reference | RV32GCP | - |
1766045 ( Base ) |
337088 ( Base ) |
19432 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | - |
1766045 ( 1.0x ) |
337090 ( 1.0 ) |
19432 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
1766045 ( 1.0x ) |
337090 ( 1.0 ) |
19432 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
1766045 ( 1.0x ) |
337090 ( 1.0 ) |
19432 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
1766045 ( 1.0x ) |
337090 ( 1.0 ) |
19432 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
1766045 ( 1.0x ) |
337090 ( 1.0 ) |
19432 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
1766045 ( 1.0x ) |
337090 ( 1.0 ) |
19432 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
572845 ( 3.1x ) |
338002 ( 1.003 ) |
19432 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | - |
454477 ( 3.9x ) |
338002 ( 1.003 ) |
19432 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | - |
395293 ( 4.5x ) |
338002 ( 1.003 ) |
19432 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | - |
366037 ( 4.8x ) |
338002 ( 1.003 ) |
19432 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | - |
362353 ( 4.9x ) |
338002 ( 1.003 ) |
19432 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | - |
360469 ( 4.9x ) |
338002 ( 1.003 ) |
19432 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | - |
1602465 ( 1.1x ) |
349230 ( 1.036 ) |
19380 ( 0.997 ) |
0 | muRISCV-NN | Scalar | RV32GCP | - |
948713 ( 1.9x ) |
350820 ( 1.041 ) |
19380 ( 0.997 ) |
0 | muRISCV-NN | Packed | RV32GCP | - |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Kernels | Mode | Arch | Auto-Vectorization |
|---|---|---|---|---|---|---|---|
495266208 ( 0.1x ) |
406111 ( 0.958 ) |
134520 ( 1.0 ) |
0 | TFLM | Reference | RV32GC | - |
495266207 ( 0.1x ) |
406117 ( 0.958 ) |
134520 ( 1.0 ) |
128 | TFLM | Reference | RV32GCV | Loop+SLP |
495266207 ( 0.1x ) |
406117 ( 0.958 ) |
134520 ( 1.0 ) |
256 | TFLM | Reference | RV32GCV | Loop+SLP |
495266207 ( 0.1x ) |
406117 ( 0.958 ) |
134520 ( 1.0 ) |
512 | TFLM | Reference | RV32GCV | Loop+SLP |
495266207 ( 0.1x ) |
406117 ( 0.958 ) |
134520 ( 1.0 ) |
1024 | TFLM | Reference | RV32GCV | Loop+SLP |
495266207 ( 0.1x ) |
406117 ( 0.958 ) |
134520 ( 1.0 ) |
2048 | TFLM | Reference | RV32GCV | Loop+SLP |
495266207 ( 0.1x ) |
406117 ( 0.958 ) |
134520 ( 1.0 ) |
4096 | TFLM | Reference | RV32GCV | Loop+SLP |
445892345 ( 0.1x ) |
418478 ( 0.987 ) |
134464 ( 1.0 ) |
0 | TFLM | Reference | RV32GCP | - |
49672979 ( Base ) |
424014 ( Base ) |
134528 ( Base ) |
0 | muRISCV-NN | Scalar | RV32GC | - |
49672979 ( 1.0x ) |
424016 ( 1.0 ) |
134528 ( 1.0 ) |
128 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
49672979 ( 1.0x ) |
424016 ( 1.0 ) |
134528 ( 1.0 ) |
256 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
49672979 ( 1.0x ) |
424016 ( 1.0 ) |
134528 ( 1.0 ) |
512 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
49672979 ( 1.0x ) |
424016 ( 1.0 ) |
134528 ( 1.0 ) |
1024 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
49672979 ( 1.0x ) |
424016 ( 1.0 ) |
134528 ( 1.0 ) |
2048 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
49672979 ( 1.0x ) |
424016 ( 1.0 ) |
134528 ( 1.0 ) |
4096 | muRISCV-NN | Scalar | RV32GCV | Loop+SLP |
13470309 ( 3.7x ) |
425198 ( 1.003 ) |
134528 ( 1.0 ) |
128 | muRISCV-NN | Vector | RV32GCV | - |
10141681 ( 4.9x ) |
425198 ( 1.003 ) |
134528 ( 1.0 ) |
256 | muRISCV-NN | Vector | RV32GCV | - |
8852559 ( 5.6x ) |
425198 ( 1.003 ) |
134528 ( 1.0 ) |
512 | muRISCV-NN | Vector | RV32GCV | - |
8348060 ( 6.0x ) |
425198 ( 1.003 ) |
134528 ( 1.0 ) |
1024 | muRISCV-NN | Vector | RV32GCV | - |
8299446 ( 6.0x ) |
425198 ( 1.003 ) |
134528 ( 1.0 ) |
2048 | muRISCV-NN | Vector | RV32GCV | - |
8302835 ( 6.0x ) |
425198 ( 1.003 ) |
134528 ( 1.0 ) |
4096 | muRISCV-NN | Vector | RV32GCV | - |
40746587 ( 1.2x ) |
435582 ( 1.027 ) |
134472 ( 1.0 ) |
0 | muRISCV-NN | Scalar | RV32GCP | - |
49175639 ( 1.0x ) |
437854 ( 1.033 ) |
134472 ( 1.0 ) |
0 | muRISCV-NN | Packed | RV32GCP | - |
Original data
Click here to download the raw files for this benchmark.