Performance Test - wom-ai/inference_results_v0.5 GitHub Wiki

Contents

Accuracy Comparision between Models

SSD Inception V2 SSD MobilenetV1 SSDLite MobilenetV1 SSD MobilenetV2 SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300 300x300
mAP[^0] 22.0%[^1] 21.6%[^2] 19.3%[^1] 18.8%[^2] 22.2%[^3] Nan 22.1%[^3]
YoloV3 YoloV3 YoloV3 YoloV3
Input Size 288x288 320x320 416x416 608x608
mAP[^0] Nan (27%?) 28.2%[^5] 31.0%[^5] 33.0%[^5]

[^0]: evaluated on COCO test 2015 (81,434 images) https://www.tensorflow.org/datasets/catalog/coco

[^1]: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

[^2]: Speed/accuracy trade-offs for modern convolutional object detectors

[^3]: MobileNetV2: Inverted Residuals and Linear Bottlenecks

[^4]: YOLOv3: An Incremental Improvement

[^5]: YOLOv4: Optimal Speed and Accuracy of Object Detection

Performance Without Preprocessing for COCO 91 Classes

Xavier

Comparision between Models (2020-06-22)

SSD Inception V2 SSD MobilenetV1 SSD MobilenetV2 SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 195.40 (5.117 ms) 681.86 (1.466 ms) 285.04 (3.508 ms) 487.60 (2.050 ms)
kernel_size in box_predictor 3 1 3 3
mAP on coco val 2017 (5,000) 27.554% 22.911% 24.305% 23.689%

Comparision along batch_size

01. ssd_mobilenet_v1_coco_2018_01_28 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 681.86 (1.466 ms) 498.855 (2.004 ms) 335.51 (2.980 ms) 202.64 (4.934 ms)

02. ssdlite_mobilenet_v2_coco_2018_05_09 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 487.60 (2.050 ms) 377.835 (2.646 ms) 252.53 (3.959 ms) 154.09 (6.489 ms)

03. ssd_inception_v2_coco_2018_01_28 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 195.40 (5.117 ms) 141.21 (7.081 ms) 92.02 (10.866 ms) 56.60 (17.665 ms)

Geforce RTX 2080 Ti

Comparision between Models (2020-02-04)

SSD Inception V2 SSD MobilenetV1 SSD MobilenetV2 SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
kernel_size in box_predictor 3 1 3 3
FPS (Latency) 500.89 (1.996ms) 1456.14 (0.68ms) 667.89 (1.497ms) 1087.70 (0.919ms)
mAP on coco val 2017 (5,000) 27.554% 22.911% 24.305% 23.689%

Comparision along batch_size

01. ssd_mobilenet_v1_coco_2018_01_28 (2020-06-11)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 1392.17 (0.718 ms) 1163.38 (0.859 ms) 949.595 (1.053 ms) 668.80 (1.495 ms)

02. ssdlite_mobilenet_v2_coco_2018_05_09 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 1083.37 (0.923 ms) 947.97 (1.054 ms) 773.55 (1.292 ms) 547.395 (1.826 ms)

03. ssd_inception_v2_coco_2018_01_28 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 503.25 (1.987 ms) 398.895 (2.506 ms) 335.02 (2.984 ms) 262.31 (3.812 ms)

Geforce GTX 1080 Ti

Comparision between Models

SSD Inception V2 SSD MobilenetV1 SSD MobilenetV2 SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 385.31 (2.595ms) 842.59 (1.186ms) 514.85 (1.942ms) 733.56 (1.363ms)
mAP on coco val 2017 (5,000) 27.664% 22.908% 24.434% 23.725%

Comparision along batch_size

01. ssd_mobilenet_v1_coco_2018_01_28 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 887.80 (1.126 ms) 721.96 (1.385 ms) 532.83 (1.876 ms) 388.29 (2.575 ms)

02. ssdlite_mobilenet_v2_coco_2018_05_09 (2020-06-16)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 757.02 (1.320 ms) 610.96 (1.636 ms) 482.04 (2.074 ms) 345.21 (2.896 ms)

03. ssd_inception_v2_coco_2018_01_28 (2020-06-22)

batch_size=1 batch_size=2 batch_size=4 batch_size=8
Platform TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
FPS (Latency) 381.81 (2.619 ms) 276.95 (3.610 ms) 216.25 (4.624 ms) 155.96 (6.411 ms)

Performance Without Preprocessing for Ball

Xavier

Comparision between Models (2020-06-23)

SSD MobilenetV1 SSD MobilenetV1 SSDLite MobilenetV1 - SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300 300x300
Platform TRT TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
kernel_size in box_predictor 1 3 3 - 3
FPS (Latency) 704.29 (1.419 ms) 425.38 (2.350 ms) 656.25 (1.523 ms) - (- ms) 500.60 (1.997 ms)

Geforce RTX 2080 Ti

Comparision between Models

SSD MobilenetV1 SSD MobilenetV1 SSDLite MobilenetV1 - SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300 300x300
Platform TRT TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
kernel_size in box_predictor 1 3 3 - 3
FPS (Latency) 1754.52 (0.569 ms) 797.24 (1.254 ms) 1653.78 (0.604 ms) - (- ms) 1100.42 (0.908 ms)

Geforce GTX 1080 Ti

Comparision between Models

SSD MobilenetV1 SSD MobilenetV1 SSDLite MobilenetV1 - SSDLite MobilenetV2
Input Size 300x300 300x300 300x300 300x300 300x300
Platform TRT TRT TRT TRT TRT
Format INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4 INT8 CHW4
kernel_size in box_predictor 1 3 3 - 3
FPS (Latency) 1061.08 (0.942 ms) 666.44 (1.500 ms) 1050.48 (0.951 ms) - (- ms) 755.02 (1.324 ms)