Benchmarks 2024 11 18 TVM LLVM Os spike_rv64 - tum-ei-eda/muriscv-nn GitHub Wiki
Setup
Simulator
- Spike (
riscv-isa-sim) (ISS, CPI=1)- Spike :
0bc176b3fca43560b9e8586cdbc41cfde073e17a - Spike PK :
7e9b671c0415dfd7b562ac934feb9380075d4aa2
- Spike :
Toolchains
-
RISC-V GCC:
- Scalar:
riscv32-unknown-elf-gcc (gc891d8dc23e) 13.2.0 - Vector:
riscv32-unknown-elf-gcc (gc891d8dc23e) 13.2.0 - Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
- Scalar:
-
LLVM/Clang:
clang version 18.1.8 (https://github.com/llvm/llvm-project.git 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)- Linker: lld (TODO)
Models
-
MLPerfTiny Benchmark
-
TODO: others!
Frameworks
-
MLonMCU :
develop -
TVM :
Nightly Pre-Build
Miscellaneous
- Used
-Osflag for compilation. - Benchmarks generated using MLonMCU deployment tool with minimal efforts.
- Memory metrics are reported in Bytes
Results (Framework: tvm, Backend: tvmaot, Toolchain: llvm, Flags: -Os, Target: spike_rv64 )
Audio Wake Words (aww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Layout | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|---|
28356244 ( 0.6x ) |
102320 ( 1.214 ) |
61360 ( 2.915 ) |
0 | NCHW | TVM | Fallback | RV64GC | 0 | - |
24009555 ( 0.7x ) |
97374 ( 1.155 ) |
61280 ( 2.911 ) |
0 | NHWC | TVM | Fallback | RV64GC | 0 | - |
5890684 ( 2.7x ) |
101312 ( 1.202 ) |
61288 ( 2.912 ) |
128 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
5890684 ( 2.7x ) |
101312 ( 1.202 ) |
61288 ( 2.912 ) |
256 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
5890684 ( 2.7x ) |
101312 ( 1.202 ) |
61288 ( 2.912 ) |
512 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
5890684 ( 2.7x ) |
101312 ( 1.202 ) |
61288 ( 2.912 ) |
1024 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
5890684 ( 2.7x ) |
101312 ( 1.202 ) |
61288 ( 2.912 ) |
2048 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
5892549 ( 2.7x ) |
101312 ( 1.202 ) |
61288 ( 2.912 ) |
4096 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
9587015 ( 1.6x ) |
98426 ( 1.167 ) |
61280 ( 2.911 ) |
128 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
7929015 ( 2.0x ) |
98426 ( 1.167 ) |
61280 ( 2.911 ) |
256 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
7100015 ( 2.2x ) |
98426 ( 1.167 ) |
61280 ( 2.911 ) |
512 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
6685515 ( 2.3x ) |
98426 ( 1.167 ) |
61280 ( 2.911 ) |
1024 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
6685515 ( 2.3x ) |
98426 ( 1.167 ) |
61280 ( 2.911 ) |
2048 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
6685515 ( 2.3x ) |
98426 ( 1.167 ) |
61280 ( 2.911 ) |
4096 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
15705400 ( Base ) |
84318 ( Base ) |
21048 ( Base ) |
0 | NHWC | muRISCV-NN | Scalar | RV64GC | 0 | - |
7141242 ( 2.2x ) |
87406 ( 1.037 ) |
21048 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
6265589 ( 2.5x ) |
87406 ( 1.037 ) |
21048 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
5827885 ( 2.7x ) |
87406 ( 1.037 ) |
21048 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
5610898 ( 2.8x ) |
87406 ( 1.037 ) |
21048 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
5610898 ( 2.8x ) |
87406 ( 1.037 ) |
21048 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
5614628 ( 2.8x ) |
87406 ( 1.037 ) |
21048 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
6022685 ( 2.6x ) |
85608 ( 1.015 ) |
25512 ( 1.212 ) |
128 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4888629 ( 3.2x ) |
85608 ( 1.015 ) |
25512 ( 1.212 ) |
256 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4330333 ( 3.6x ) |
85608 ( 1.015 ) |
25512 ( 1.212 ) |
512 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4291202 ( 3.7x ) |
85608 ( 1.015 ) |
25512 ( 1.212 ) |
1024 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4291202 ( 3.7x ) |
85608 ( 1.015 ) |
25512 ( 1.212 ) |
2048 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4293067 ( 3.7x ) |
85608 ( 1.015 ) |
25512 ( 1.212 ) |
4096 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
Image Classification (resnet)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Layout | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|---|
123430647 ( 0.5x ) |
210362 ( 1.596 ) |
110208 ( 1.921 ) |
0 | NCHW | TVM | Fallback | RV64GC | 0 | - |
100169350 ( 0.6x ) |
203596 ( 1.545 ) |
110208 ( 1.921 ) |
0 | NHWC | TVM | Fallback | RV64GC | 0 | - |
24209026 ( 2.3x ) |
209638 ( 1.59 ) |
110208 ( 1.921 ) |
128 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24209026 ( 2.3x ) |
209638 ( 1.59 ) |
110208 ( 1.921 ) |
256 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24209027 ( 2.3x ) |
209640 ( 1.59 ) |
110208 ( 1.921 ) |
512 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24209027 ( 2.3x ) |
209640 ( 1.59 ) |
110208 ( 1.921 ) |
1024 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24209028 ( 2.3x ) |
209642 ( 1.59 ) |
110208 ( 1.921 ) |
2048 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24210893 ( 2.3x ) |
209642 ( 1.59 ) |
110208 ( 1.921 ) |
4096 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
35938112 ( 1.6x ) |
204460 ( 1.551 ) |
110208 ( 1.921 ) |
128 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
26886718 ( 2.1x ) |
204448 ( 1.551 ) |
110208 ( 1.921 ) |
256 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24622016 ( 2.3x ) |
204460 ( 1.551 ) |
110208 ( 1.921 ) |
512 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24177792 ( 2.3x ) |
204460 ( 1.551 ) |
110208 ( 1.921 ) |
1024 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24177790 ( 2.3x ) |
204448 ( 1.551 ) |
110208 ( 1.921 ) |
2048 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24177790 ( 2.3x ) |
204448 ( 1.551 ) |
110208 ( 1.921 ) |
4096 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
55742014 ( Base ) |
131818 ( Base ) |
57368 ( Base ) |
0 | NHWC | muRISCV-NN | Scalar | RV64GC | 0 | - |
25746948 ( 2.2x ) |
135698 ( 1.029 ) |
57368 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
17783799 ( 3.1x ) |
135698 ( 1.029 ) |
57368 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
14051179 ( 4.0x ) |
135698 ( 1.029 ) |
57368 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
12352622 ( 4.5x ) |
135698 ( 1.029 ) |
57368 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
11525047 ( 4.8x ) |
135698 ( 1.029 ) |
57368 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
10904832 ( 5.1x ) |
135698 ( 1.029 ) |
57368 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
15256232 ( 3.7x ) |
133748 ( 1.015 ) |
57368 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
9673032 ( 5.8x ) |
133748 ( 1.015 ) |
57368 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
7127192 ( 7.8x ) |
133748 ( 1.015 ) |
57368 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
5885064 ( 9.5x ) |
133748 ( 1.015 ) |
57368 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4958665 ( 11.2x ) |
133748 ( 1.015 ) |
57368 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
4711438 ( 11.8x ) |
133748 ( 1.015 ) |
57368 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
Anomaly Detection (toycar)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Layout | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|---|
3197040 ( 0.5x ) |
576294 ( 1.857 ) |
7400 ( 1.121 ) |
0 | NCHW | TVM | Fallback | RV64GC | 0 | - |
3197040 ( 0.5x ) |
576294 ( 1.857 ) |
7400 ( 1.121 ) |
0 | NHWC | TVM | Fallback | RV64GC | 0 | - |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
128 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
256 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
512 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
1024 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
2048 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
4096 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
128 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
256 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
512 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
1024 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
2048 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1036307 ( 1.6x ) |
575490 ( 1.855 ) |
7400 ( 1.121 ) |
4096 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
1667784 ( Base ) |
310308 ( Base ) |
6600 ( Base ) |
0 | NHWC | muRISCV-NN | Scalar | RV64GC | 0 | - |
628314 ( 2.7x ) |
311064 ( 1.002 ) |
6600 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
510873 ( 3.3x ) |
311062 ( 1.002 ) |
6600 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
452153 ( 3.7x ) |
311062 ( 1.002 ) |
6600 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
422794 ( 3.9x ) |
311068 ( 1.002 ) |
6600 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
408112 ( 4.1x ) |
311060 ( 1.002 ) |
6600 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
406273 ( 4.1x ) |
311062 ( 1.002 ) |
6600 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
620215 ( 2.7x ) |
311778 ( 1.005 ) |
6600 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
501847 ( 3.3x ) |
311778 ( 1.005 ) |
6600 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
442662 ( 3.8x ) |
311776 ( 1.005 ) |
6600 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
413407 ( 4.0x ) |
311778 ( 1.005 ) |
6600 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
409721 ( 4.1x ) |
311774 ( 1.005 ) |
6600 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
407839 ( 4.1x ) |
311778 ( 1.005 ) |
6600 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
Visual Wake Words (vww)
| Cycles (Speedup) | Total ROM (rel.) | Total RAM (rel.) | VLEN | Layout | Kernels | Mode | Arch | Unroll | Auto-Vectorization |
|---|---|---|---|---|---|---|---|---|---|
81861282 ( 0.5x ) |
534248 ( 1.684 ) |
182968 ( 2.091 ) |
0 | NCHW | TVM | Fallback | RV64GC | 0 | - |
69691432 ( 0.6x ) |
516610 ( 1.628 ) |
182808 ( 2.089 ) |
0 | NHWC | TVM | Fallback | RV64GC | 0 | - |
18260567 ( 2.4x ) |
529648 ( 1.669 ) |
182832 ( 2.089 ) |
128 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
18260564 ( 2.4x ) |
529644 ( 1.669 ) |
182832 ( 2.089 ) |
256 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
18260565 ( 2.4x ) |
529644 ( 1.669 ) |
182832 ( 2.089 ) |
512 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
18260567 ( 2.4x ) |
529648 ( 1.669 ) |
182832 ( 2.089 ) |
1024 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
18260564 ( 2.4x ) |
529642 ( 1.669 ) |
182832 ( 2.089 ) |
2048 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
18262431 ( 2.4x ) |
529646 ( 1.669 ) |
182832 ( 2.089 ) |
4096 | NCHW | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
28854119 ( 1.5x ) |
519382 ( 1.637 ) |
182808 ( 2.089 ) |
128 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
24202200 ( 1.8x ) |
519384 ( 1.637 ) |
182808 ( 2.089 ) |
256 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
21996047 ( 2.0x ) |
519384 ( 1.637 ) |
182808 ( 2.089 ) |
512 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
21070381 ( 2.1x ) |
519384 ( 1.637 ) |
182808 ( 2.089 ) |
1024 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
20704332 ( 2.1x ) |
519384 ( 1.637 ) |
182808 ( 2.089 ) |
2048 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
20678261 ( 2.1x ) |
519384 ( 1.637 ) |
182808 ( 2.089 ) |
4096 | NHWC | TVM | Fallback | RV64GCV | 0 | Loop+SLP |
44077978 ( Base ) |
317302 ( Base ) |
87504 ( Base ) |
0 | NHWC | muRISCV-NN | Scalar | RV64GC | 0 | - |
18469688 ( 2.4x ) |
321316 ( 1.013 ) |
87504 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
15492695 ( 2.8x ) |
321314 ( 1.013 ) |
87504 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
14066535 ( 3.1x ) |
321314 ( 1.013 ) |
87504 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13542327 ( 3.3x ) |
321310 ( 1.013 ) |
87504 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13328581 ( 3.3x ) |
321314 ( 1.013 ) |
87504 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13312112 ( 3.3x ) |
321312 ( 1.013 ) |
87504 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Scalar | RV64GCV | 0 | Loop+SLP |
13478670 ( 3.3x ) |
319544 ( 1.007 ) |
87504 ( 1.0 ) |
128 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
10157613 ( 4.3x ) |
319540 ( 1.007 ) |
87504 ( 1.0 ) |
256 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
8866158 ( 5.0x ) |
319544 ( 1.007 ) |
87504 ( 1.0 ) |
512 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
8364019 ( 5.3x ) |
319544 ( 1.007 ) |
87504 ( 1.0 ) |
1024 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
8316244 ( 5.3x ) |
319544 ( 1.007 ) |
87504 ( 1.0 ) |
2048 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
8318109 ( 5.3x ) |
319544 ( 1.007 ) |
87504 ( 1.0 ) |
4096 | NHWC | muRISCV-NN | Vector | RV64GCV | 0 | - |
Original data
Click here to download the raw files for this benchmark.