How to evaluate accuracy and speed of YOLOv4 - Sudhakar17/darknet GitHub Wiki

How to evaluate accuracy and speed of YOLOv4

How to evaluate AP of YOLOv4 on the MS COCO evaluation server

0. Set width=608 height=608 (or 512x512, or 416x416) in the [net] section of cfg/yolov4.cfg file https://github.com/AlexeyAB/darknet/blob/6f718c257815a984253346bba8fb7aa756c55090/cfg/yolov4.cfg#L8-L9
1. Download and unzip test-dev2017 dataset from MS COCO server: http://images.cocodataset.org/zips/test2017.zip
2. Download list of images for Detection taks and replace the paths with yours (to the unzipped images from test2017.zip): https://raw.githubusercontent.com/AlexeyAB/darknet/master/scripts/testdev2017.txt
3. Download yolov4.weights file: https://drive.google.com/open?id=1cewMfusmPjYWbrnuJRuKhPMwRe_b9PaT
4. Content of the file cfg/coco.data should be

classes= 80
train  = <replace with your path>/trainvalno5k.txt
valid = <replace with your path>/testdev2017.txt
names = data/coco.names
backup = backup
eval=coco

5. Create /results/ folder near with ./darknet executable file
6. Run validation: ./darknet detector valid cfg/coco.data cfg/yolov4.cfg yolov4.weights
7. Rename the file /results/coco_results.json to detections_test-dev2017_yolov4_results.json and compress it to detections_test-dev2017_yolov4_results.zip
8. Submit file detections_test-dev2017_yolov4_results.zip to the MS COCO evaluation server for the test-dev2019 (bbox) https://competitions.codalab.org/competitions/20794#participate

You will get such results (AP=0.435 and AP50=0.657) in the end of file View scoring output log

overall performance
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.435
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.657
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.473
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.267
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.467
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.533
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.342
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.549
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.580
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.403
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.617
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.713
Done (t=334.58s)

How to evaluate FPS of YOLOv4 on GPU

1. Compile Darknet with GPU=1 CUDNN=1 CUDNN_HALF=1 OPENCV=1 in the Makefile (or use the same settings with Cmake)
2. Download yolov4.weights file: https://drive.google.com/open?id=1cewMfusmPjYWbrnuJRuKhPMwRe_b9PaT
3. Get any .avi/.mp4 video file (preferably not more than 1920x1080 to avoid bottlenecks in CPU performance)
4. Run one of two commands and look at the AVG FPS:

• include video_capturing + NMS + drawing_bboxes: ./darknet detector demo cfg/coco.data cfg/yolov4.cfg yolov4.weights test.mp4 -dont_show -ext_output
• exclude video_capturing + NMS + drawing_bboxes: ./darknet detector demo cfg/coco.data cfg/yolov4.cfg yolov4.weights test.mp4 -benchmark

Accuracy and FPS on RTX 2070 (R) and Tesla V100 (V):

• yolov4.cfg - 245 MB: yolov4.weights
  • width=608 height=608 in cfg: 65.7% [email protected] (43.5% [email protected]:0.95) - 34(R) FPS / 62(V) FPS - 128.5 BFlops
  • width=512 height=512 in cfg: 64.9% [email protected] (43.0% [email protected]:0.95) - 45(R) FPS / 83(V) FPS - 91.1 BFlops
  • width=416 height=416 in cfg: 62.8% [email protected] (41.2% [email protected]:0.95) - 55(R) FPS / 96(V) FPS - 60.1 BFlops
  • width=320 height=320 in cfg: 60% [email protected] ( 38% [email protected]:0.95) - 63(R) FPS / 123(V) FPS - 35.5 BFlops

---

Result - paper: https://arxiv.org/abs/2004.10934

---