00. experiments_reproduce_east - bluekingsong/vision

- # reproduce east in handwritten date
EAST:
$python calc_detect_pr.py single_east_data/valid_gt.txt EAST/checkpoint_path/model.ckpt-99991.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=1243/1357=0.916 recall=1243/1289=0.964
iou=0.5 filed_name=handwritten_date precision=1235/1357=0.910 recall=1235/1289=0.958
iou=0.7 filed_name=handwritten_date precision=1155/1357=0.851 recall=1155/1289=0.896
iou=0.9 filed_name=handwritten_date precision=334/1357=0.246 recall=334/1289=0.259

all exps based on exp_v00.

exp_v00: try to reproduce east performance by df, failed, TODO: check reason
$python calc_detect_pr.py single_east_data/valid_gt.txt ckp_v00/model.ckpt-80001.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=1238/1405=0.881 recall=1238/1289=0.960
iou=0.5 filed_name=handwritten_date precision=1225/1405=0.872 recall=1225/1289=0.950
iou=0.7 filed_name=handwritten_date precision=1145/1405=0.815 recall=1145/1289=0.888
iou=0.9 filed_name=handwritten_date precision=341/1405=0.243 recall=341/1289=0.265

exp_v01: study batch_size effect, set batch_size=1, conclusion: has effect
$python calc_detect_pr.py single_east_data/valid_gt.txt ckp_v01/model.ckpt-800001.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=949/1044=0.909 recall=949/1289=0.736
iou=0.5 filed_name=handwritten_date precision=836/1044=0.801 recall=836/1289=0.649
iou=0.7 filed_name=handwritten_date precision=668/1044=0.640 recall=668/1289=0.518
iou=0.9 filed_name=handwritten_date precision=85/1044=0.081 recall=85/1289=0.066

exp_v02: study random scale augmentation effect, remove scale data augmentation. conclusion: no effect
$python calc_detect_pr.py single_east_data/valid_gt.txt ckp_v02/model.ckpt-80001.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=1238/1405=0.881 recall=1238/1289=0.960
iou=0.5 filed_name=handwritten_date precision=1224/1405=0.871 recall=1224/1289=0.950
iou=0.7 filed_name=handwritten_date precision=1148/1405=0.817 recall=1148/1289=0.891
iou=0.9 filed_name=handwritten_date precision=312/1405=0.222 recall=312/1289=0.242

exp_v03: reproduce by data_provider
$python calc_detect_pr.py single_east_data/valid_gt.txt ckp_v03/model.ckpt-100001.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=1202/1280=0.939 recall=1202/1289=0.933
iou=0.5 filed_name=handwritten_date precision=1191/1280=0.930 recall=1191/1289=0.924
iou=0.7 filed_name=handwritten_date precision=1098/1280=0.858 recall=1098/1289=0.852
iou=0.9 filed_name=handwritten_date precision=227/1280=0.177 recall=227/1289=0.176

exp_v04: set batch_size=8(exp_v00 batch size = 16), conclusion:

some more detail

exp_v01: study batch_size effect, set batch_size=1, conclusion: has effect
VALID at 390000 {’east_geo_loss’: 0.005632227063179016, ‘east_mask_loss’: 0.006088011264801026, ‘east_loss’: 0.01172023892402649, ‘cnt’: 1.0}
—

step=400000 2018-12-30 08:14:18.349914 {’east_geo_loss’: 0.003285013198852539, ‘east_mask_loss’: 0.0018535886764526367, ‘east_loss’: 0.005138599014282226}

VALID at 400000 {’east_geo_loss’: 0.00591195285320282, ‘east_mask_loss’: 0.003051222264766693, ‘east_loss’: 0.008963173627853394, ‘cnt’: 1.0}
—
step=410000 2018-12-30 08:27:49.336675 {’east_geo_loss’: 0.0033724605560302733, ‘east_mask_loss’: 0.0018801816940307618, ‘east_loss’: 0.005252634429931641}

2018-12-30 08:27:50,132 WARNING data_provider.py 469: data starving.
VALID at 410000 {’east_geo_loss’: 0.008191832900047302, ‘east_mask_loss’: 0.003619641363620758, ‘east_loss’: 0.011811474561691284, ‘cnt’: 1.0}
—

step=420000 2018-12-30 08:41:23.617459 {’east_geo_loss’: 0.0033055057525634766, ‘east_mask_loss’: 0.0018376970291137695, ‘east_loss’: 0.005143213653564453}

VALID at 420000 {’east_geo_loss’: 0.004661917090415955, ‘east_mask_loss’: 0.0036114728450775146, ‘east_loss’: 0.00827338993549347, ‘cnt’: 1.0}
—

step=430000 2018-12-30 08:54:46.949476 {’east_geo_loss’: 0.003204813003540039, ‘east_mask_loss’: 0.0018153650283813476, ‘east_loss’: 0.0050201774597167965}

VALID at 430000 {’east_geo_loss’: 0.004903594553470611, ‘east_mask_loss’: 0.0047979578375816345, ‘east_loss’: 0.009701550602912903, ‘cnt’: 1.0}
—

step=440000 2018-12-30 09:08:18.736077 {’east_geo_loss’: 0.003254217529296875, ‘east_mask_loss’: 0.0018461009979248047, ‘east_loss’: 0.005100307083129882}

VALID at 440000 {’east_geo_loss’: 0.004795103073120117, ‘east_mask_loss’: 0.0040751898288726805, ‘east_loss’: 0.00887029230594635, ‘cnt’: 1.0}
—

step=450000 2018-12-30 09:21:34.907261 {’east_geo_loss’: 0.0032248409271240235, ‘east_mask_loss’: 0.0018254030227661132, ‘east_loss’: 0.0050502368927001955}

VALID at 450000 {’east_geo_loss’: 0.006923761367797852, ‘east_mask_loss’: 0.0066301429271698, ‘east_loss’: 0.013553909063339233, ‘cnt’: 1.0}
—

step=460000 2018-12-30 09:34:59.741860 {’east_geo_loss’: 0.0032838172912597655, ‘east_mask_loss’: 0.0018264083862304687, ‘east_loss’: 0.005110225677490234}

VALID at 460000 {’east_geo_loss’: 0.006746258735656738, ‘east_mask_loss’: 0.0036584192514419554, ‘east_loss’: 0.010404679775238037, ‘cnt’: 1.0}
—

step=470000 2018-12-30 09:48:18.459105 {’east_geo_loss’: 0.0031822065353393557, ‘east_mask_loss’: 0.001823769187927246, ‘east_loss’: 0.005005967712402344}

VALID at 470000 {’east_geo_loss’: 0.005151717662811279, ‘east_mask_loss’: 0.004571780562400818, ‘east_loss’: 0.009723497033119201, ‘cnt’: 1.0}
—

step=480000 2018-12-30 10:01:58.074687 {’east_geo_loss’: 0.003311049270629883, ‘east_mask_loss’: 0.001809821891784668, ‘east_loss’: 0.005120865249633789}

VALID at 480000 {’east_geo_loss’: 0.006182377934455871, ‘east_mask_loss’: 0.003761570453643799, ‘east_loss’: 0.00994394838809967, ‘cnt’: 1.0}
—

step=490000 2018-12-30 10:15:22.921657 {’east_geo_loss’: 0.0032574718475341796, ‘east_mask_loss’: 0.0018431367874145509, ‘east_loss’: 0.005100617980957031}

VALID at 490000 {’east_geo_loss’: 0.005792948603630066, ‘east_mask_loss’: 0.004317229688167572, ‘east_loss’: 0.010110175609588623, ‘cnt’: 1.0}
—

step=500000 2018-12-30 10:28:48.674708 {’east_geo_loss’: 0.003249636459350586, ‘east_mask_loss’: 0.001844031524658203, ‘east_loss’: 0.005093660736083984}

VALID at 500000 {’east_geo_loss’: 0.00509069561958313, ‘east_mask_loss’: 0.003789078891277313, ‘east_loss’: 0.008879776000976563, ‘cnt’: 1.0}
—

step=510000 2018-12-30 10:42:12.203391 {’east_geo_loss’: 0.0032330360412597655, ‘east_mask_loss’: 0.0018449411392211913, ‘east_loss’: 0.005077980804443359}

VALID at 510000 {’east_geo_loss’: 0.005523521304130554, ‘east_mask_loss’: 0.003920042216777802, ‘east_loss’: 0.009443563222885133, ‘cnt’: 1.0}
—

step=520000 2018-12-30 10:55:45.125299 {’east_geo_loss’: 0.0032460186004638673, ‘east_mask_loss’: 0.0018181514739990235, ‘east_loss’: 0.0050641735076904296}

VALID at 520000 {’east_geo_loss’: 0.00911930799484253, ‘east_mask_loss’: 0.003817085325717926, ‘east_loss’: 0.012936394214630127, ‘cnt’: 1.0}
—

step=530000 2018-12-30 11:09:11.284778 {’east_geo_loss’: 0.0032005260467529296, ‘east_mask_loss’: 0.0017874088287353515, ‘east_loss’: 0.004987932968139649}

VALID at 530000 {’east_geo_loss’: 0.007280945777893066, ‘east_mask_loss’: 0.003669172525405884, ‘east_loss’: 0.010950119495391845, ‘cnt’: 1.0}
—

step=540000 2018-12-30 11:22:41.049785 {’east_geo_loss’: 0.003225849151611328, ‘east_mask_loss’: 0.0017744691848754883, ‘east_loss’: 0.005000312423706054}

VALID at 540000 {’east_geo_loss’: 0.0054806554317474365, ‘east_mask_loss’: 0.004301155209541321, ‘east_loss’: 0.00978181004524231, ‘cnt’: 1.0}
—

step=550000 2018-12-30 11:35:58.334048 {’east_geo_loss’: 0.003161687660217285, ‘east_mask_loss’: 0.001815271759033203, ‘east_loss’: 0.00497694091796875}

VALID at 550000 {’east_geo_loss’: 0.006198288798332214, ‘east_mask_loss’: 0.0041217350959777835, ‘east_loss’: 0.01032002568244934, ‘cnt’: 1.0}
—

step=560000 2018-12-30 11:49:38.018968 {’east_geo_loss’: 0.0032445732116699218, ‘east_mask_loss’: 0.0018120874404907227, ‘east_loss’: 0.0050566596984863285}

VALID at 560000 {’east_geo_loss’: 0.009688826203346252, ‘east_mask_loss’: 0.003279511332511902, ‘east_loss’: 0.01296833872795105, ‘cnt’: 1.0}
—

step=570000 2018-12-30 12:03:04.772532 {’east_geo_loss’: 0.003132486343383789, ‘east_mask_loss’: 0.0017932140350341798, ‘east_loss’: 0.00492570686340332}

VALID at 570000 {’east_geo_loss’: 0.006153633594512939, ‘east_mask_loss’: 0.003017667829990387, ‘east_loss’: 0.009171301126480102, ‘cnt’: 1.0}
—

step=580000 2018-12-30 12:16:36.975257 {’east_geo_loss’: 0.0032413787841796874, ‘east_mask_loss’: 0.001826521110534668, ‘east_loss’: 0.005067891693115234}

VALID at 580000 {’east_geo_loss’: 0.005610930919647217, ‘east_mask_loss’: 0.00313414067029953, ‘east_loss’: 0.008745073676109313, ‘cnt’: 1.0}
—

step=590000 2018-12-30 12:30:02.538661 {’east_geo_loss’: 0.0032442211151123045, ‘east_mask_loss’: 0.0018215764999389649, ‘east_loss’: 0.005065801239013672}

VALID at 590000 {’east_geo_loss’: 0.005577406287193299, ‘east_mask_loss’: 0.004243394434452057, ‘east_loss’: 0.00982079803943634, ‘cnt’: 1.0}
—

step=600000 2018-12-30 12:43:25.544832 {’east_geo_loss’: 0.003228879165649414, ‘east_mask_loss’: 0.0017944841384887696, ‘east_loss’: 0.005023360824584961}

VALID at 600000 {’east_geo_loss’: 0.00839380443096161, ‘east_mask_loss’: 0.003468806743621826, ‘east_loss’: 0.01186261534690857, ‘cnt’: 1.0}
—

step=610000 2018-12-30 12:56:51.149789 {’east_geo_loss’: 0.003189530372619629, ‘east_mask_loss’: 0.0017908426284790038, ‘east_loss’: 0.004980362319946289}

VALID at 610000 {’east_geo_loss’: 0.00492099404335022, ‘east_mask_loss’: 0.004713878333568573, ‘east_loss’: 0.009634873867034911, ‘cnt’: 1.0}
—

step=620000 2018-12-30 13:10:13.799570 {’east_geo_loss’: 0.003115163803100586, ‘east_mask_loss’: 0.0017588388442993164, ‘east_loss’: 0.004874001312255859}

VALID at 620000 {’east_geo_loss’: 0.009838317036628724, ‘east_mask_loss’: 0.004344536662101745, ‘east_loss’: 0.014182853698730468, ‘cnt’: 1.0}
—

step=630000 2018-12-30 13:23:37.210990 {’east_geo_loss’: 0.003143817710876465, ‘east_mask_loss’: 0.0017901369094848632, ‘east_loss’: 0.0049339488983154295}

VALID at 630000 {’east_geo_loss’: 0.006855733990669251, ‘east_mask_loss’: 0.0037861642241477965, ‘east_loss’: 0.010641900300979614, ‘cnt’: 1.0}
—

step=640000 2018-12-30 13:37:05.906227 {’east_geo_loss’: 0.0032222309112548827, ‘east_mask_loss’: 0.0017967487335205077, ‘east_loss’: 0.005018993759155274}

VALID at 640000 {’east_geo_loss’: 0.008809560537338256, ‘east_mask_loss’: 0.002912079393863678, ‘east_loss’: 0.011721640825271606, ‘cnt’: 1.0}
—

step=650000 2018-12-30 13:50:44.656772 {’east_geo_loss’: 0.0031996904373168943, ‘east_mask_loss’: 0.0017712867736816406, ‘east_loss’: 0.0049709686279296875}

VALID at 650000 {’east_geo_loss’: 0.008305248618125916, ‘east_mask_loss’: 0.004379176497459412, ‘east_loss’: 0.012684423923492432, ‘cnt’: 1.0}
—

step=660000 2018-12-30 14:04:03.178428 {’east_geo_loss’: 0.0030915149688720703, ‘east_mask_loss’: 0.0017882591247558594, ‘east_loss’: 0.004879772567749023}

VALID at 660000 {’east_geo_loss’: 0.013310343027114868, ‘east_mask_loss’: 0.003011299669742584, ‘east_loss’: 0.016321635246276854, ‘cnt’: 1.0}
—

step=670000 2018-12-30 14:17:43.061143 {’east_geo_loss’: 0.003149821090698242, ‘east_mask_loss’: 0.0017626346588134765, ‘east_loss’: 0.004912465286254883}

VALID at 670000 {’east_geo_loss’: 0.005611932873725891, ‘east_mask_loss’: 0.0051006418466568, ‘east_loss’: 0.010712577104568482, ‘cnt’: 1.0}
—

step=680000 2018-12-30 14:31:12.031838 {’east_geo_loss’: 0.003088619041442871, ‘east_mask_loss’: 0.0017354766845703124, ‘east_loss’: 0.004824075698852539}

VALID at 680000 {’east_geo_loss’: 0.00936779499053955, ‘east_mask_loss’: 0.004653623700141907, ‘east_loss’: 0.014021419286727906, ‘cnt’: 1.0}
—

step=690000 2018-12-30 14:44:33.307759 {’east_geo_loss’: 0.0031571306228637694, ‘east_mask_loss’: 0.0017826087951660157, ‘east_loss’: 0.00493974380493164}

VALID at 690000 {’east_geo_loss’: 0.012709583044052125, ‘east_mask_loss’: 0.003283202350139618, ‘east_loss’: 0.015992785692214965, ‘cnt’: 1.0}
—
step=700000 2018-12-30 14:58:15.886346 {’east_geo_loss’: 0.003163726997375488, ‘east_mask_loss’: 0.001782762336730957, ‘east_loss’: 0.004946478652954101}

2018-12-30 14:58:17,349 WARNING data_provider.py 469: data starving.
VALID at 700000 {’east_geo_loss’: 0.009450974464416504, ‘east_mask_loss’: 0.0052706170082092284, ‘east_loss’: 0.014721591472625733, ‘cnt’: 1.0}
—

step=710000 2018-12-30 15:11:47.517949 {’east_geo_loss’: 0.003154188346862793, ‘east_mask_loss’: 0.0017666399002075196, ‘east_loss’: 0.004920817565917969}

VALID at 710000 {’east_geo_loss’: 0.006585944294929504, ‘east_mask_loss’: 0.006277841329574585, ‘east_loss’: 0.012863789796829223, ‘cnt’: 1.0}
—

step=720000 2018-12-30 15:25:16.562206 {’east_geo_loss’: 0.0031420969009399416, ‘east_mask_loss’: 0.001756956100463867, ‘east_loss’: 0.004899049758911133}

VALID at 720000 {’east_geo_loss’: 0.008829094171524048, ‘east_mask_loss’: 0.005084836483001709, ‘east_loss’: 0.013913933038711548, ‘cnt’: 1.0}
—

step=730000 2018-12-30 15:38:51.640139 {’east_geo_loss’: 0.003155385398864746, ‘east_mask_loss’: 0.001775503158569336, ‘east_loss’: 0.0049308818817138675}

VALID at 730000 {’east_geo_loss’: 0.005114814043045044, ‘east_mask_loss’: 0.004226581752300262, ‘east_loss’: 0.009341395497322082, ‘cnt’: 1.0}
—

step=740000 2018-12-30 15:52:26.380466 {’east_geo_loss’: 0.0031831308364868166, ‘east_mask_loss’: 0.0017443546295166017, ‘east_loss’: 0.004927476119995117}

VALID at 740000 {’east_geo_loss’: 0.005414391756057739, ‘east_mask_loss’: 0.005026752948760986, ‘east_loss’: 0.010441144704818725, ‘cnt’: 1.0}
—

step=750000 2018-12-30 16:05:51.579404 {’east_geo_loss’: 0.003093899345397949, ‘east_mask_loss’: 0.0017546728134155273, ‘east_loss’: 0.004848561096191406}

VALID at 750000 {’east_geo_loss’: 0.005263662338256836, ‘east_mask_loss’: 0.004010825157165527, ‘east_loss’: 0.009274490475654602, ‘cnt’: 1.0}
—

step=760000 2018-12-30 16:19:25.624386 {’east_geo_loss’: 0.003150834083557129, ‘east_mask_loss’: 0.0017772092819213867, ‘east_loss’: 0.004928048324584961}

VALID at 760000 {’east_geo_loss’: 0.008888566493988037, ‘east_mask_loss’: 0.0035751903057098387, ‘east_loss’: 0.012463754415512085, ‘cnt’: 1.0}
—

step=770000 2018-12-30 16:32:41.775649 {’east_geo_loss’: 0.0031755645751953125, ‘east_mask_loss’: 0.0017818050384521484, ‘east_loss’: 0.004957379531860351}

VALID at 770000 {’east_geo_loss’: 0.007303702831268311, ‘east_mask_loss’: 0.004703494608402252, ‘east_loss’: 0.012007200717926025, ‘cnt’: 1.0}
—

step=780000 2018-12-30 16:46:23.919490 {’east_geo_loss’: 0.0032832298278808593, ‘east_mask_loss’: 0.0017792390823364258, ‘east_loss’: 0.005062472915649414}

VALID at 780000 {’east_geo_loss’: 0.00609509527683258, ‘east_mask_loss’: 0.004618191123008728, ‘east_loss’: 0.0107132887840271, ‘cnt’: 1.0}
—
step=790000 2018-12-30 17:00:03.582231 {’east_geo_loss’: 0.003146574783325195, ‘east_mask_loss’: 0.0017663461685180665, ‘east_loss’: 0.00491291275024414}

2018-12-30 17:00:04,281 WARNING data_provider.py 469: data starving.
VALID at 790000 {’east_geo_loss’: 0.005578851103782654, ‘east_mask_loss’: 0.0035122403502464293, ‘east_loss’: 0.009091092348098755, ‘cnt’: 1.0}
—
step=800000 2018-12-30 17:13:32.786428 {’east_geo_loss’: 0.0031922582626342775, ‘east_mask_loss’: 0.001811439323425293, ‘east_loss’: 0.005003701782226562}

2018-12-30 17:13:33,060 WARNING data_provider.py 469: data starving.
VALID at 800000 {’east_geo_loss’: 0.004858759641647339, ‘east_mask_loss’: 0.002880642414093018, ‘east_loss’: 0.0077394026517868045, ‘cnt’: 1.0}

exp_v02: study random scale augmentation effect, remove scale data augmentation
$python calc_detect_pr.py single_east_data/valid_gt.txt ckp_v02/model.ckpt-50001.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=1239/1442=0.859 recall=1239/1289=0.961
iou=0.5 filed_name=handwritten_date precision=1230/1442=0.853 recall=1230/1289=0.954
iou=0.7 filed_name=handwritten_date precision=1149/1442=0.797 recall=1149/1289=0.891
iou=0.9 filed_name=handwritten_date precision=327/1442=0.227 recall=327/1289=0.254
$python calc_detect_pr.py single_east_data/valid_gt.txt ckp_v02/model.ckpt-60001.txt
usage: gt_text_filename pred_text_filename [score_thres]
iou=0.3 filed_name=handwritten_date precision=1238/1410=0.878 recall=1238/1289=0.960
iou=0.5 filed_name=handwritten_date precision=1228/1410=0.871 recall=1228/1289=0.953
iou=0.7 filed_name=handwritten_date precision=1138/1410=0.807 recall=1138/1289=0.883
iou=0.9 filed_name=handwritten_date precision=326/1410=0.231 recall=326/1289=0.253

exp_v03: reproduce by data_provider

exp_v04: set batch_size=8

00. experiments_reproduce_east - bluekingsong/vision_material GitHub Wiki

⚠️ GitHub.com Fallback ⚠️

00. experiments_reproduce_east - bluekingsong/vision_material GitHub Wiki

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️