Evaluation - mikel-brostrom/Yolov5_DeepSort_Pytorch Wiki

How to evaluate

  1. Download this model, trained on the crowd-human-dataset, and place it under Yolov5_DeepSort_Pytorch/yolov5/weights/

  2. Run the evaluation script

./MOT16_eval/eval.sh  

NOTICE! This MOT16 evaluation is performed on the train split, NOT the test split, as the test ground truth is not publicly available. However, this is not an issue as the train dataset is actually never used for training.

Obtained metrics:


CLEAR: ch_yolov5m_deep_sort-pedestrianMOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag      
MOT16-02                           33.887    77.114    34.397    38.109    91.124    16.667    38.889    44.444    25.165    6796      11037     662       91        9         21        24        198       
MOT16-04                           63.831    76.326    63.997    72.149    89.848    40.964    39.759    19.277    46.75     34312     13245     3877      79        34        33        16        369       
MOT16-05                           56.307    76.523    57.334    71.414    83.531    40        49.6      10.4      39.541    4869      1949      960       70        50        62        13        150       
MOT16-09                           62.507    81.999    63.401    77.268    84.784    52        40        8         48.598    4062      1195      729       47        13        10        2         70        
MOT16-10                           52.703    75.324    53.239    59.685    90.253    27.778    46.296    25.926    37.975    7352      4966      794       66        15        25        14        272       
MOT16-11                           64.138    84.251    64.465    79.191    84.32     50.725    34.783    14.493    51.666    7265      1909      1351      30        35        24        10        83        
MOT16-13                           30.332    68.89     30.847    41.956    79.065    9.3458    52.336    38.318    17.279    4804      6646      1272      59        10        56        41        281       
COMBINED                           53.776    76.957    54.177    62.913    87.807    32.108    44.681    23.211    39.28     69460     40947     9645      442       166       231       120       1423      

Identity: ch_yolov5m_deep_sort-pedestrianIDF1      IDR       IDP       IDTP      IDFN      IDFP      
MOT16-02                           36.361    25.784    61.652    4598      13235     2860      
MOT16-04                           67.341    60.708    75.6      28871     18686     9318      
MOT16-05                           39.583    36.712    42.94     2503      4315      3326      
MOT16-09                           50.378    48.145    52.828    2531      2726      2260      
MOT16-10                           54.251    45.064    68.144    5551      6767      2595      
MOT16-11                           47.768    46.316    49.315    4249      4925      4367      
MOT16-13                           39.393    30.148    56.814    3452      7998      2624      
COMBINED                           54.619    46.877    65.426    51755     58652     27350     

Count: ch_yolov5m_deep_sort-pedestrianDets      GT_Dets   IDs       GT_IDs    
MOT16-02                           7458      17833     50        54        
MOT16-04                           38189     47557     99        83        
MOT16-05                           5829      6818      42        125       
MOT16-09                           4791      5257      21        25        
MOT16-10                           8146      12318     46        54        
MOT16-11                           8616      9174      49        69        
MOT16-13                           6076      11450     54        107       
COMBINED                           79105     110407    361       517


Crowd-human Yolov5m Deep Sort compared to other online trackers

Tracker MOTA IDF1 MT ML IDs FPS
EAMTT 52.5 53.3 19.9% 34.9% 910 <5.5
SORTwHPD16 59.8 53.8 25.4% 22.7% 1423 <8.6
DeepSORT2 61.4 62.2 32.8% 18.2% 781 <6.4
RAR16wVGG 63.0 63.8 39.9% 22.1% 482 <1.4
VMaxx 62.6 49.2 32.7% 21.1% 1389 <3.9
TubeTK 64.0 59.4 33.5% 19.4% 1117 1.0
JDE 64.4 55.8 35.4% 20.0% 1544 18.5
TAP 64.8 73.5 38.5% 21.6% 571 <8.0
CNNMTT 65.2 62.2 32.4% 21.3% 946 <5.3
POI 66.1 65.1 34.0% 20.8% 805 <5.0
CTrackerV1 67.6 57.2 32.9% 23.1% 1897 6.8
FairMOT 74.9 72.8 44.7% 15.9% 1074 25.9
yolov5m_crowdhuman+deep_sort (conf_thres=0.5) 53.8 54.6 23.7% 17.0% 361 <21 (Nvidia quadro P2000)
yolov5m_crowdhuman+deep_sort (conf_thres=0.4) 51.6 53.8 25.1% 13.0% 422 <21 (Nvidia quadro P2000)

We achieve state of the art results in ML (Ratio of mostly lost trajectories) and IDs (Number of identity switches). By hovering over the metric in this table you can read about them.

Thanks to zengjie617789 for helping out with this evaluation!