Training with Kraken - UB-Mannheim/AustrianNewspapers GitHub Wiki
Trained models are available from https://ub-backup.bib.uni-mannheim.de/~stweil/tesstrain/kraken/.
It was tested on 2021-11-18 with latest AustrianNewspapers and Kraken from GitHub. Prepare the lists of PAGE XML files for training and evaluation:
ls TrainingSet_ONB_Newseye_GT_M1+/*xml >list.train
ls ValidationSet_ONB_Newseye_GT_M1+/*xml >list.eval
A test with 64 GiB RAM and --preload fails early:
time nice ketos train -f page -t list.train -e list.eval -o austriannewspapers -d cuda:0 --preload --threads 32 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
Building training set [#-----------------------------------] 2110/53980 00:13:09 [36.8371] Text line "" is empty after transformations
Building training set [#-----------------------------------] 2847/53980 00:14:16 [52.2694] No boundary given for line
Building training set [##----------------------------------] 3929/53980 00:14:37 [73.5322] No boundary given for line
Building training set [##----------------------------------] 3977/53980 00:14:39 [74.6955] No boundary given for line
Building training set [###---------------------------------] 5600/53980 00:15:40 [114.1772] Text line "" is empty after transformations
Building training set [####--------------------------------] 6695/53980 00:16:42 [147.5183] Text line "" is empty after transformations
Building training set [####--------------------------------] 6784/53980 00:16:44 [149.6519] Text line "" is empty after transformations
Building training set [####--------------------------------] 6785/53980 00:16:44 [149.7190] Text line "" is empty after transformations
Building training set [####--------------------------------] 6791/53980 00:16:47 [150.0795] Text line "" is empty after transformations
[150.1191] Text line "" is empty after transformations
Building training set [####--------------------------------] 6797/53980 00:16:47 [150.3513] Text line "" is empty after transformations
Building training set [####--------------------------------] 6798/53980 00:16:46 [150.4424] Text line "" is empty after transformations
Building training set [####--------------------------------] 6804/53980 00:16:46 [150.8330] Text line "" is empty after transformations
[150.8335] Text line "" is empty after transformations
Building training set [####--------------------------------] 6807/53980 00:16:49 [150.9921] Text line "" is empty after transformations
[151.0413] Text line "" is empty after transformations
Building training set [####--------------------------------] 6815/53980 00:16:49 [151.5072] Text line "" is empty after transformations
[151.5077] Text line "" is empty after transformations
Building training set [####--------------------------------] 6820/53980 00:16:49 [151.8785] Text line "" is empty after transformations
Building training set [####--------------------------------] 6836/53980 00:16:52 [152.1064] Text line "" is empty after transformations
[152.1065] Text line "" is empty after transformations
[152.2932] Text line "" is empty after transformations
[152.2971] Text line "" is empty after transformations
Building training set [####--------------------------------] 6844/53980 00:16:51 [152.6193] Text line "" is empty after transformations
Building training set [####--------------------------------] 6847/53980 00:16:51 [152.7810] Text line "" is empty after transformations
[152.8248] Text line "" is empty after transformations
Building training set [####--------------------------------] 6851/53980 00:16:51 [153.0566] Text line "" is empty after transformations
Building training set [####--------------------------------] 6854/53980 00:16:55 [153.1822] Text line "" is empty after transformations
Building training set [####--------------------------------] 7300/53980 00:17:45 [172.5671] Text line "" is empty after transformations
Building training set [#####-------------------------------] 7863/53980 00:18:31 [195.3467] Text line "" is empty after transformations
Building training set [#####-------------------------------] 8603/53980 00:19:06 [220.7014] Text line "" is empty after transformations
Building training set [######------------------------------] 10314/53980 00:19:35 [283.3722] Text line "" is empty after transformations
Building training set [#######-----------------------------] 11875/53980 00:20:54 [359.6720] Text line "" is empty after transformations
[359.7049] Text line "" is empty after transformations
Building training set [########----------------------------] 12191/53980 00:21:13 [377.1624] Text line "" is empty after transformations
[377.1626] Text line "" is empty after transformations
Building training set [########----------------------------] 12194/53980 00:21:13 [377.3784] Text line "" is empty after transformations
Building training set [#########---------------------------] 13573/53980 00:22:10 [452.6622] Text line "" is empty after transformations
Building training set [##########--------------------------] 15360/53980 00:23:31 [566.7300] Text line "" is empty after transformations
Building training set [##########--------------------------] 15382/53980 00:23:32 [568.2695] Text line "" is empty after transformations
[568.2697] Text line "" is empty after transformations
Building training set [##########--------------------------] 15395/53980 00:23:31 [568.9404] Text line "" is empty after transformations
[...]
Building training set [########################------------] 37135/53980 00:22:47 [3021.7851] Text line "" is empty after transformations
Building training set [##########################----------] 39063/53980 00:21:13 [3339.9409] Text line "" is empty after transformations
Building training set [##########################----------] 39075/53980 00:21:12 [3341.6370] Text line "" is empty after transformations
Building training set [##########################----------] 39105/53980 00:21:10 [3346.4863] Text line "" is empty after transformations
Building training set [##########################----------] 39109/53980 00:21:10 [3347.5681] Text line "" is empty after transformations
Building training set [##########################----------] 39196/53980 00:21:05 [3362.2098] Text line "" is empty after transformations
Building training set [##########################----------] 39435/53980 00:20:52 [3402.4703] Text line "" is empty after transformations
Building training set [#############################-------] 43515/53980 00:16:33 Traceback (most recent call last):
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/bin/ketos", line 10, in <module>
sys.exit(cli())
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/ketos.py", line 569, in train
trainer = KrakenTrainer.recognition_train_gen(hyper_params,
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/train.py", line 679, in recognition_train_gen
for im in pool.imap_unordered(partial(_star_fun, gt_set.parse), training_data, 5):
File "/usr/lib/python3.9/multiprocessing/pool.py", line 448, in <genexpr>
return (item for chunk in result for item in chunk)
File "/usr/lib/python3.9/multiprocessing/pool.py", line 870, in next
raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[{'text': 'Kornmehl fein Nr. 2 8 fl. 59 kr., β β 9 kr.', 'image': tensor([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]), 'baseline': [(194, 639), (342, 639), (420, 640), (497, 638), (541, 638), (808, 637), (859, 636), (924, 635), (975, 634), (1047, 639), (1157, 638), (1239, 633), (1310, 633)], 'boundary': [(194, 601), (1310, 601), (1310, 648), (194, 648)], 'im_mode': <built-in method mode of Tensor object at 0x7f350ccc1ef0>, 'preload': True, 'preparse': True}, {'text': 'SΓ€cke werden ΕΏeparat berechnet und im guten ZuΕΏtande franco zum gleichen', 'image': tensor([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]), 'baseline': [(173, 689), (260, 688), (391, 687), (523, 686), (685, 686), (776, 685), (834, 684), (940, 684), (1101, 682), (1217, 682), (1300, 680), (1436, 678)], 'boundary': [(173, 648), (1436, 648), (1436, 697), (173, 697)], 'im_mode': <built-in method mode of Tensor object at 0x7f346c369770>, 'preload': True, 'preparse': True}, {'text': 'PreiΕΏe zurΓΌckgenommen. 23', 'image': tensor([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]), 'baseline': [(92, 730), (471, 725), (1413, 720), (1419, 725), (1434, 727)], 'boundary': [(93, 688), (1434, 688), (1434, 747), (93, 747)], 'im_mode': <built-in method mode of Tensor object at 0x7f346c369e50>, 'preload': True, 'preparse': True}, {'text': 'VerΕΏteigerungsβΈKundmachung.', 'image': tensor([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]), 'baseline': [(349, 816), (1176, 808)], 'boundary': [(349, 751), (1178, 751), (1178, 829), (349, 829)], 'im_mode': <built-in method mode of Tensor object at 0x7f346c3697c0>, 'preload': True, 'preparse': True}, {'text': 'Am 4. Juli d. Js. um 9 Uhr frΓΌh angefangen, werden im HauΕΏe Nr. 57', 'image': tensor([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]), 'baseline': [(172, 868), (231, 869), (282, 868), (372, 868), (418, 867), (497, 864), (568, 865), (610, 865), (692, 863), (804, 862), (973, 861), (1122, 861), (1180, 861), (1297, 859), (1373, 859), (1436, 858)], 'boundary': [(174, 826), (1436, 826), (1436, 878), (174, 878)], 'im_mode': <built-in method mode of Tensor object at 0x7f346c3692c0>, 'preload': True, 'preparse': True}]'. Reason: 'RuntimeError('falseINTERNAL ASSERT FAILED at "../aten/src/ATen/MapAllocator.cpp":300, please report a bug to PyTorch. unable to write to file </torch_3110641_1359>')'
real 69m3,582s
user 628m1,017s
sys 224m17,866s
The 2nd test with modified command line (no --preload), 64 GiB RAM and 8 GiB swap fails with out of memory after stage 1 during the validation:
time nice ketos train -f page -t list.train -e list.eval -o austriannewspapers -d cuda:0 --threads 32 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
Building training set [------------------------------------] 1193/53980 00:00:44[5.4199] Text line "" is empty after transformations
Building training set [------------------------------------] 1220/53980 00:00:44[5.4601] Text line "" is empty after transformations
Building training set [------------------------------------] 1288/53980 00:00:44[5.5276] Text line "" is empty after transformations
Building training set [------------------------------------] 1348/53980 00:00:44[5.5837] Text line "" is empty after transformations
Building training set [------------------------------------] 1349/53980 00:00:44[5.5856] Text line "" is empty after transformations
Building training set [------------------------------------] 1359/53980 00:00:44[5.5915] Text line "" is empty after transformations
[5.5919] Text line "" is empty after transformations
Building training set [------------------------------------] 1442/53980 00:00:44[5.6532] Text line "" is empty after transformations
Building training set [------------------------------------] 1445/53980 00:00:44[5.6568] Text line "" is empty after transformations
Building training set [------------------------------------] 1447/53980 00:00:44[5.6592] Text line "" is empty after transformations
Building training set [##----------------------------------] 3804/53980 00:00:42[6.0189] Text line "" is empty after transformations
Building training set [##----------------------------------] 4088/53980 00:00:42[6.2008] Text line "" is empty after transformations
[6.2011] Text line "" is empty after transformations
Building training set [##----------------------------------] 4301/53980 00:00:41[6.3401] Text line "" is empty after transformations
Building training set [###---------------------------------] 4645/53980 00:00:32[6.6525] Text line "" is empty after transformations
Building training set [###---------------------------------] 4662/53980 00:00:32[6.6656] Text line "" is empty after transformations
Building training set [###---------------------------------] 4702/53980 00:00:32[6.6964] Text line "" is empty after transformations
Building training set [###---------------------------------] 4716/53980 00:00:32[6.7079] Text line "" is empty after transformations
Building training set [###---------------------------------] 4845/53980 00:00:32[6.8163] Text line "" is empty after transformations
Building training set [####--------------------------------] 6465/53980 00:00:31[9.6205] Text line "" is empty after transformations
Building training set [####--------------------------------] 7310/53980 00:00:34[11.6840] Text line "" is empty after transformations
Building training set [######------------------------------] 9325/53980 00:00:53[17.7361] Text line "" is empty after transformations
Building training set [######------------------------------] 9660/53980 00:00:56[18.9291] Text line "" is empty after transformations
Building training set [##########--------------------------] 15720/53980 00:01:47[50.8938] Text line "" is empty after transformations
Building training set [###########-------------------------] 17410/53980 00:02:00[63.6588] Text line "" is empty after transformations
[63.6592] Text line "" is empty after transformations
Building training set [###########-------------------------] 17415/53980 00:02:00[63.7153] Text line "" is empty after transformations
[63.7155] Text line "" is empty after transformations
Building training set [#############-----------------------] 19810/53980 00:02:14[84.1980] Text line "" is empty after transformations
Building training set [#############-----------------------] 20215/53980 00:02:15[87.9396] Text line "" is empty after transformations
Building training set [#############-----------------------] 20260/53980 00:02:15[88.3330] Text line "" is empty after transformations
Building training set [#############-----------------------] 20430/53980 00:02:17[89.9166] Text line "" is empty after transformations
Building training set [#############-----------------------] 20505/53980 00:02:17[90.6151] Text line "" is empty after transformations
Building training set [##############----------------------] 21410/53980 00:02:21[99.4895] Text line "" is empty after transformations
[99.4896] Text line "" is empty after transformations
[99.4897] Text line "" is empty after transformations
Building training set [##############----------------------] 21415/53980 00:02:21[99.5623] Text line "" is empty after transformations
[99.5624] Text line "" is empty after transformations
[99.5624] Text line "" is empty after transformations
[99.5625] Text line "" is empty after transformations
[99.5625] Text line "" is empty after transformations
Building training set [##############----------------------] 21425/53980 00:02:21[99.5884] Text line "" is empty after transformations
Building training set [##############----------------------] 22105/53980 00:02:24[106.5605] Text line "" is empty after transformations
Building training set [##############----------------------] 22195/53980 00:02:24[107.5499] Text line "" is empty after transformations
Building training set [##############----------------------] 22230/53980 00:02:24[107.9135] Text line "" is empty after transformations
Building training set [##############----------------------] 22415/53980 00:02:25[109.7858] Text line "" is empty after transformations
Building training set [###############---------------------] 22740/53980 00:02:27[113.3687] Text line "" is empty after transformations
Building training set [###############---------------------] 23710/53980 00:02:29[123.6277] Text line "" is empty after transformations
Building training set [###############---------------------] 23715/53980 00:02:29[123.6914] Text line "" is empty after transformations
Building training set [##################------------------] 27400/53980 00:02:43[174.9981] Text line "" is empty after transformations
Building training set [##################------------------] 28185/53980 00:02:45[187.2889] Text line "" is empty after transformations
[187.2891] Text line "" is empty after transformations
Building training set [###################-----------------] 28865/53980 00:02:46[198.3607] Text line "" is empty after transformations
Building training set [####################----------------] 30690/53980 00:02:49[229.3225] No boundary given for line
Building training set [#####################---------------] 31775/53980 00:02:49[248.7394] No boundary given for line
Building training set [#####################---------------] 31830/53980 00:02:49[249.7704] No boundary given for line
Building training set [#######################-------------] 34535/53980 00:02:45[300.2663] Text line "" is empty after transformations
Building training set [#########################-----------] 38285/53980 00:02:31[376.4473] Text line "" is empty after transformations
[376.4474] Text line "" is empty after transformations
[376.5285] Text line "" is empty after transformations
Building training set [#########################-----------] 38390/53980 00:02:31[378.7820] Text line "" is empty after transformations
Building training set [#########################-----------] 38415/53980 00:02:30[379.3506] Text line "" is empty after transformations
Building training set [#########################-----------] 38845/53980 00:02:29[388.8674] Text line "" is empty after transformations
Building training set [##########################----------] 39100/53980 00:02:27[394.4711] Text line "" is empty after transformations
[394.4713] Text line "" is empty after transformations
Building training set [##########################----------] 40090/53980 00:02:22[416.7219] Text line "" is empty after transformations
[416.7220] Text line "" is empty after transformations
Building training set [############################--------] 42265/53980 00:02:12[486.7281] Text line "" is empty after transformations
Building training set [############################--------] 42340/53980 00:02:12[489.9111] Text line "" is empty after transformations
[489.9112] Text line "" is empty after transformations
Building training set [############################--------] 42350/53980 00:02:12[490.0899] Text line "" is empty after transformations
Building training set [############################--------] 42360/53980 00:02:12[490.4746] Text line "" is empty after transformations
Building training set [############################--------] 42365/53980 00:02:12[490.6931] Text line "" is empty after transformations
Building training set [############################--------] 42370/53980 00:02:12[490.8640] Text line "" is empty after transformations
Building training set [############################--------] 42380/53980 00:02:12[491.2308] Text line "" is empty after transformations
[491.2309] Text line "" is empty after transformations
Building training set [############################--------] 42385/53980 00:02:12[491.6978] Text line "" is empty after transformations
Building training set [############################--------] 42400/53980 00:02:12[492.0886] Text line "" is empty after transformations
Building training set [############################--------] 42405/53980 00:02:12[492.2673] Text line "" is empty after transformations
[492.2674] Text line "" is empty after transformations
Building training set [############################--------] 42415/53980 00:02:12[492.7701] Text line "" is empty after transformations
Building training set [############################--------] 42425/53980 00:02:12[493.0587] Text line "" is empty after transformations
[493.0588] Text line "" is empty after transformations
Building training set [############################--------] 42435/53980 00:02:12[493.4583] Text line "" is empty after transformations
Building training set [############################--------] 42440/53980 00:02:12[493.7949] Text line "" is empty after transformations
[493.8640] Text line "" is empty after transformations
Building training set [############################--------] 42450/53980 00:02:12[494.0369] Text line "" is empty after transformations
Building training set [############################--------] 42455/53980 00:02:12[494.2519] Text line "" is empty after transformations
Building training set [############################--------] 42460/53980 00:02:12[494.5990] Text line "" is empty after transformations
[494.5992] Text line "" is empty after transformations
Building training set [############################--------] 43050/53980 00:02:09[517.9345] Text line "" is empty after transformations
Building training set [##############################------] 46015/53980 00:01:49[637.7727] Text line "" is empty after transformations
Building training set [###############################-----] 47535/53980 00:01:34[701.6049] Text line "" is empty after transformations
Building training set [#################################---] 49825/53980 00:01:06[801.3096] Text line "" is empty after transformations
Building training set [##################################--] 52005/53980 00:00:33[898.7751] Text line "" is empty after transformations
Building training set [##################################--] 52020/53980 00:00:33[899.4902] Text line "" is empty after transformations
[899.4903] Text line "" is empty after transformations
[899.4903] Text line "" is empty after transformations
Building training set [####################################] 53980/53980
Building validation set [####--------------------------------] 564/4813[990.4600] Text line "" is empty after transformations
Building validation set [#####-------------------------------] 678/4813[990.5313] Text line "" is empty after transformations
Building validation set [#####-------------------------------] 802/4813[990.6373] Text line "" is empty after transformations
Building validation set [####################################] 4813/4813 [993.2221] alphabet mismatch: chars in training set only: {'Ε', 'Ι', '~', 'Ε', 'β΅', '⬀', 'β
', 'β
', 'β
', 'β
', 'β³', 'Γ', 'β²', 'Κ', 'βΈ', 'β
', 'ΕΎ', 'β', 'Γ«', 'β°', 'βΌ', 'β', 'β
', 'βΉ', 'β
', 'Ε ', 'β', 'β€', 'Γ', 'Β±', 'β', 'Γ²', 'Γͺ', 'β ', 'βΈ«', 'β―', 'Γ³', 'Β³', 'Γ', 'Γ’', 'Γ±', 'β', 'Γ¦', 'Γ»', 'βΆ', 'β', 'β', 'β', 'ΒΉ', 'β', 'β·', 'β΄', 'Β°', 'Γ΄', 'β
', 'οΌ'} (not included in accuracy test during training)
[993.2223] alphabet mismatch: chars in validation set only: {'Ε'} (not trained)
Initializing model β
stage 1/β [####################################] 53882/53882
Traceback (most recent call last):
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/bin/ketos", line 10, in <module>
sys.exit(cli())
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/ketos.py", line 604, in train
trainer.run(_print_eval, _draw_progressbar)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/train.py", line 498, in run
eval_res = self.evaluator(self.model, self.val_set, self.device)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/train.py", line 373, in recognition_evaluator_fn
chars, error = compute_error(rec, val_loader)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/dataset.py", line 265, in compute_error
preds = model.predict_string(batch['image'], batch['seq_lens'])
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/models.py", line 116, in predict_string
o, olens = self.forward(line, lens)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/models.py", line 79, in forward
o, olens = self.nn.nn(line, lens)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/layers.py", line 27, in forward
inputs = module(*inputs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3137907) is killed by signal: Killed.
real 72m27,371s
user 311m7,803s
sys 274m42,800s
For the 3rd test, the available memory was increased. 64 GiB RAM und 40 GiB swap seems to be sufficient and are nearly filled completely during validation at the end of a stage. Training requires about an hour per epoch.
time nice ketos train -f page -t list.train -e list.eval -o austriannewspapers -d cuda:0 --threads 32 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
[...]
Initializing model β
stage 1/β [####################################] 53882/53882 Accuracy report (1) 0.9592 171710 7000
stage 2/β [####################################] 53882/53882 Accuracy report (2) 0.9715 171689 4890
stage 3/β [####################################] 53882/53882 Accuracy report (3) 0.9742 171834 4425
stage 4/β [####################################] 53882/53882 Accuracy report (4) 0.9792 171575 3570
stage 5/β [####################################] 53882/53882 Accuracy report (5) 0.9801 171715 3414
stage 6/β [####################################] 53882/53882 Accuracy report (6) 0.9816 171639 3163
stage 7/β [####################################] 53882/53882 Accuracy report (7) 0.9824 171679 3024
stage 8/β [####################################] 53882/53882 Accuracy report (8) 0.9828 171697 2961
stage 9/β [####################################] 53882/53882 Accuracy report (9) 0.9830 171636 2921
stage 10/β [####################################] 53882/53882 Accuracy report (10) 0.9838 171772 2790
stage 11/β [####################################] 53882/53882 Accuracy report (11) 0.9842 171759 2711
stage 12/β [####################################] 53882/53882 Accuracy report (12) 0.9843 171793 2699
stage 13/β [####################################] 53882/53882 Accuracy report (13) 0.9829 171695 2943
stage 14/β [####################################] 53882/53882 Accuracy report (14) 0.9847 171794 2620
stage 15/β [####################################] 53882/53882 Accuracy report (15) 0.9844 171733 2683
stage 16/β [####################################] 53882/53882 Accuracy report (16) 0.9854 171912 2508
stage 17/β [####################################] 53882/53882 Accuracy report (17) 0.9850 171734 2584
stage 18/β [####################################] 53882/53882 Accuracy report (18) 0.9850 171754 2570
stage 19/β [####################################] 53882/53882 Accuracy report (19) 0.9849 171822 2595
stage 20/β [####################################] 53882/53882 Accuracy report (20) 0.9844 171761 2687
stage 21/β [####################################] 53882/53882 Accuracy report (21) 0.9850 171839 2576
stage 22/β [####################################] 53882/53882 Accuracy report (22) 0.9852 171711 2545
stage 23/β [####################################] 53882/53882 Accuracy report (23) 0.9850 171831 2581
stage 24/β [####################################] 53882/53882 Accuracy report (24) 0.9855 171714 2493
stage 25/β [####################################] 53882/53882 Accuracy report (25) 0.9851 171610 2554
stage 26/β [####################################] 53882/53882 Accuracy report (26) 0.9850 171660 2570
stage 27/β [####################################] 53882/53882 Accuracy report (27) 0.9856 171819 2479
stage 28/β [####################################] 53882/53882 Accuracy report (28) 0.9855 171747 2495
stage 29/β [####################################] 53882/53882 Accuracy report (29) 0.9850 171765 2577
stage 30/β [####################################] 53882/53882 Accuracy report (30) 0.9851 171795 2559
stage 31/β [####################################] 53882/53882 Accuracy report (31) 0.9853 171670 2524
stage 32/β [####################################] 53882/53882 Accuracy report (32) 0.9846 171682 2644
stage 33/β [####################################] 53882/53882 Accuracy report (33) 0.9853 171741 2520
stage 34/β [####################################] 53882/53882 Accuracy report (34) 0.9856 171765 2478
stage 35/β [####################################] 53882/53882 Accuracy report (35) 0.9852 171904 2548
stage 36/β [####################################] 53882/53882 Accuracy report (36) 0.9856 171801 2473
stage 37/β [####################################] 53882/53882 Accuracy report (37) 0.9849 171932 2600
stage 38/β [####################################] 53882/53882 Accuracy report (38) 0.9854 171744 2511
stage 39/β [####################################] 53882/53882 Accuracy report (39) 0.9851 171842 2569
stage 40/β [####################################] 53882/53882 Accuracy report (40) 0.9847 171712 2620
stage 41/β [####################################] 53882/53882 Accuracy report (41) 0.9854 171709 2501
stage 42/β [####################################] 53882/53882 Accuracy report (42) 0.9853 171690 2523
stage 43/β [####################################] 53882/53882 Accuracy report (43) 0.9833 171674 2862
stage 44/β [####################################] 53882/53882 Accuracy report (44) 0.9855 171717 2498
Moving best model austriannewspapers_36.mlmodel (0.9856054186820984) to austriannewspapers_best.mlmodel
real 2261m28,446s
user 11756m21,598s
sys 12025m18,217s
ls -lt austriannewspapers_*
-rw-r--r-- 1 stweil stweil 16242901 20. Nov 10:30 austriannewspapers_best.mlmodel
-rw-r--r-- 1 stweil stweil 16243149 20. Nov 10:30 austriannewspapers_44.mlmodel
-rw-r--r-- 1 stweil stweil 16243118 20. Nov 09:40 austriannewspapers_43.mlmodel
-rw-r--r-- 1 stweil stweil 16243087 20. Nov 08:49 austriannewspapers_42.mlmodel
-rw-r--r-- 1 stweil stweil 16243056 20. Nov 07:58 austriannewspapers_41.mlmodel
-rw-r--r-- 1 stweil stweil 16243025 20. Nov 07:07 austriannewspapers_40.mlmodel
-rw-r--r-- 1 stweil stweil 16242994 20. Nov 06:16 austriannewspapers_39.mlmodel
-rw-r--r-- 1 stweil stweil 16242963 20. Nov 05:25 austriannewspapers_38.mlmodel
-rw-r--r-- 1 stweil stweil 16242932 20. Nov 04:34 austriannewspapers_37.mlmodel
-rw-r--r-- 1 stweil stweil 16242901 20. Nov 03:43 austriannewspapers_36.mlmodel
-rw-r--r-- 1 stweil stweil 16242870 20. Nov 02:52 austriannewspapers_35.mlmodel
-rw-r--r-- 1 stweil stweil 16242840 20. Nov 02:01 austriannewspapers_34.mlmodel
-rw-r--r-- 1 stweil stweil 16242809 20. Nov 01:10 austriannewspapers_33.mlmodel
-rw-r--r-- 1 stweil stweil 16242778 20. Nov 00:19 austriannewspapers_32.mlmodel
-rw-r--r-- 1 stweil stweil 16242747 19. Nov 23:28 austriannewspapers_31.mlmodel
-rw-r--r-- 1 stweil stweil 16242716 19. Nov 22:37 austriannewspapers_30.mlmodel
-rw-r--r-- 1 stweil stweil 16242685 19. Nov 21:46 austriannewspapers_29.mlmodel
-rw-r--r-- 1 stweil stweil 16242654 19. Nov 20:56 austriannewspapers_28.mlmodel
-rw-r--r-- 1 stweil stweil 16242623 19. Nov 20:05 austriannewspapers_27.mlmodel
-rw-r--r-- 1 stweil stweil 16242592 19. Nov 19:14 austriannewspapers_26.mlmodel
-rw-r--r-- 1 stweil stweil 16242561 19. Nov 18:24 austriannewspapers_25.mlmodel
-rw-r--r-- 1 stweil stweil 16242530 19. Nov 17:32 austriannewspapers_24.mlmodel
-rw-r--r-- 1 stweil stweil 16242499 19. Nov 16:41 austriannewspapers_23.mlmodel
-rw-r--r-- 1 stweil stweil 16242468 19. Nov 15:50 austriannewspapers_22.mlmodel
-rw-r--r-- 1 stweil stweil 16242437 19. Nov 14:59 austriannewspapers_21.mlmodel
-rw-r--r-- 1 stweil stweil 16242406 19. Nov 14:02 austriannewspapers_20.mlmodel
-rw-r--r-- 1 stweil stweil 16242376 19. Nov 13:12 austriannewspapers_19.mlmodel
-rw-r--r-- 1 stweil stweil 16242345 19. Nov 12:21 austriannewspapers_18.mlmodel
-rw-r--r-- 1 stweil stweil 16242315 19. Nov 11:31 austriannewspapers_17.mlmodel
-rw-r--r-- 1 stweil stweil 16242285 19. Nov 10:41 austriannewspapers_16.mlmodel
-rw-r--r-- 1 stweil stweil 16242255 19. Nov 09:50 austriannewspapers_15.mlmodel
-rw-r--r-- 1 stweil stweil 16242225 19. Nov 08:59 austriannewspapers_14.mlmodel
-rw-r--r-- 1 stweil stweil 16242195 19. Nov 08:08 austriannewspapers_13.mlmodel
-rw-r--r-- 1 stweil stweil 16242165 19. Nov 07:18 austriannewspapers_12.mlmodel
-rw-r--r-- 1 stweil stweil 16242135 19. Nov 06:28 austriannewspapers_11.mlmodel
-rw-r--r-- 1 stweil stweil 16242105 19. Nov 05:37 austriannewspapers_10.mlmodel
-rw-r--r-- 1 stweil stweil 16242074 19. Nov 04:46 austriannewspapers_9.mlmodel
-rw-r--r-- 1 stweil stweil 16242044 19. Nov 03:56 austriannewspapers_8.mlmodel
-rw-r--r-- 1 stweil stweil 16242014 19. Nov 03:05 austriannewspapers_7.mlmodel
-rw-r--r-- 1 stweil stweil 16241984 19. Nov 02:14 austriannewspapers_6.mlmodel
-rw-r--r-- 1 stweil stweil 16241954 19. Nov 01:23 austriannewspapers_5.mlmodel
-rw-r--r-- 1 stweil stweil 16241924 19. Nov 00:33 austriannewspapers_4.mlmodel
-rw-r--r-- 1 stweil stweil 16241894 18. Nov 23:41 austriannewspapers_3.mlmodel
-rw-r--r-- 1 stweil stweil 16241864 18. Nov 22:51 austriannewspapers_2.mlmodel
-rw-r--r-- 1 stweil stweil 16241834 18. Nov 22:00 austriannewspapers_1.mlmodel
stweil@ocr-02:~/src/github/UB-Mannheim/AustrianNewspapers$ source venv3.9/bin/activate
(venv3.9) stweil@ocr-02:~/src/github/UB-Mannheim/AustrianNewspapers$ time nice ketos train -f page -t list.train -e list.eval -o austriannewspapers -d cuda:0 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/20/22 21:21:06] WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
[05/20/22 21:21:07] WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
[05/20/22 21:21:08] WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
[05/20/22 21:21:09] WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING No boundary given for line train.py:351
WARNING No boundary given for line train.py:351
WARNING No boundary given for line train.py:351
WARNING Text line "" is empty after transformations train.py:351
[05/20/22 21:21:10] WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING Text line "" is empty after transformations train.py:351
WARNING alphabet mismatch: chars in training set only: {'⬀', 'Γ΄', 'β', 'β', 'β', 'Γ²', 'β―', 'β', 'Ε ', 'β
', 'β
', 'Β³', 'β
', 'β°', 'β·', 'Γ', 'β
', 'Ε', train.py:304
'βΈ', 'β
', 'βΆ', 'Γ’', 'β³', 'βΈ«', 'Γͺ', 'βΌ', 'βΉ', 'Ι', 'Γ»', 'β΄', 'β΅', 'Γ³', 'β', 'β ', 'Ε', 'β
', 'Γ¦', 'Γ«', 'β', 'β€', 'ΕΎ', 'β
', 'β', 'Γ', 'β²', 'Β°',
'Β±', 'οΌ', 'β', 'Κ', 'Γ±', 'β', 'ΒΉ', 'Γ', 'β
', '~'} (not included in accuracy test during training)
WARNING alphabet mismatch: chars in validation set only: {'Ε'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/20/22 21:21:23] WARNING Non-encodable sequence Ε... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
ββββββ³ββββββββββββ³βββββββββββββββββββββββββββ³βββββββββ
β β Name β Type β Params β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β 0 β net β MultiParamSequential β 4.1 M β
β 1 β net.C_0 β ActConv2D β 1.3 K β
β 2 β net.Do_1 β Dropout β 0 β
β 3 β net.Mp_2 β MaxPool β 0 β
β 4 β net.C_3 β ActConv2D β 40.0 K β
β 5 β net.Do_4 β Dropout β 0 β
β 6 β net.Mp_5 β MaxPool β 0 β
β 7 β net.C_6 β ActConv2D β 55.4 K β
β 8 β net.Do_7 β Dropout β 0 β
β 9 β net.Mp_8 β MaxPool β 0 β
β 10 β net.C_9 β ActConv2D β 110 K β
β 11 β net.Do_10 β Dropout β 0 β
β 12 β net.S_11 β Reshape β 0 β
β 13 β net.L_12 β TransposedSummarizingRNN β 1.9 M β
β 14 β net.Do_13 β Dropout β 0 β
β 15 β net.L_14 β TransposedSummarizingRNN β 963 K β
β 16 β net.Do_15 β Dropout β 0 β
β 17 β net.L_16 β TransposedSummarizingRNN β 963 K β
β 18 β net.Do_17 β Dropout β 0 β
β 19 β net.O_18 β LinSoftmax β 71.0 K β
ββββββ΄ββββββββββββ΄βββββββββββββββββββββββββββ΄βββββββββ
Trainable params: 4.1 M
Non-trainable params: 0
Total params: 4.1 M
Total estimated model params size (MB): 16
stage 0/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:11 val_accuracy: 0.96013 early_stopping: 0/20 0.96013
stage 1/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:35:04 val_accuracy: 0.97261 early_stopping: 0/20 0.97261
stage 2/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:39 val_accuracy: 0.97690 early_stopping: 0/20 0.97690
stage 3/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:00 val_accuracy: 0.97981 early_stopping: 0/20 0.97981
stage 4/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:56 val_accuracy: 0.97999 early_stopping: 0/20 0.97999
stage 5/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:14 val_accuracy: 0.98280 early_stopping: 0/20 0.98280
stage 6/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:07 val_accuracy: 0.98406 early_stopping: 0/20 0.98406
stage 7/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:55 val_accuracy: 0.98363 early_stopping: 1/20 0.98406
stage 8/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:37 val_accuracy: 0.98446 early_stopping: 0/20 0.98446
stage 9/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:47 val_accuracy: 0.98415 early_stopping: 1/20 0.98446
stage 10/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:20 val_accuracy: 0.97640 early_stopping: 2/20 0.98446
stage 11/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:41 val_accuracy: 0.98462 early_stopping: 0/20 0.98462
stage 12/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:46 val_accuracy: 0.98495 early_stopping: 0/20 0.98495
stage 13/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:26 val_accuracy: 0.98457 early_stopping: 1/20 0.98495
stage 14/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:48 val_accuracy: 0.98458 early_stopping: 2/20 0.98495
stage 15/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:07 val_accuracy: 0.98428 early_stopping: 3/20 0.98495
stage 16/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:10 val_accuracy: 0.98499 early_stopping: 0/20 0.98499
stage 17/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:04 val_accuracy: 0.98551 early_stopping: 0/20 0.98551
stage 18/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:53 val_accuracy: 0.98430 early_stopping: 1/20 0.98551
stage 19/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:35:13 val_accuracy: 0.98513 early_stopping: 2/20 0.98551
stage 20/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:06 val_accuracy: 0.98447 early_stopping: 3/20 0.98551
stage 21/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:54 val_accuracy: 0.98555 early_stopping: 0/20 0.98555
stage 22/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:42 val_accuracy: 0.98514 early_stopping: 1/20 0.98555
stage 23/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:35:14 val_accuracy: 0.98536 early_stopping: 2/20 0.98555
stage 24/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:29 val_accuracy: 0.98482 early_stopping: 3/20 0.98555
stage 25/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:03 val_accuracy: 0.98423 early_stopping: 4/20 0.98555
stage 26/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:55 val_accuracy: 0.98496 early_stopping: 5/20 0.98555
stage 27/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:38 val_accuracy: 0.98544 early_stopping: 6/20 0.98555
stage 28/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:47 val_accuracy: 0.98554 early_stopping: 7/20 0.98555
stage 29/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:13 val_accuracy: 0.98510 early_stopping: 8/20 0.98555
stage 30/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:29 val_accuracy: 0.98523 early_stopping: 9/20 0.98555
stage 31/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:51 val_accuracy: 0.98474 early_stopping: 10/20 0.98555
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:54 val_accuracy: 0.98462 early_stopping: 11/20 0.98555
stage 33/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:14 val_accuracy: 0.98568 early_stopping: 0/20 0.98568
stage 34/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:29 val_accuracy: 0.98521 early_stopping: 1/20 0.98568
stage 35/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:16 val_accuracy: 0.98524 early_stopping: 2/20 0.98568
stage 36/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:32 val_accuracy: 0.98502 early_stopping: 3/20 0.98568
stage 37/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:56 val_accuracy: 0.98427 early_stopping: 4/20 0.98568
stage 38/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:31 val_accuracy: 0.98456 early_stopping: 5/20 0.98568
stage 39/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:42 val_accuracy: 0.98524 early_stopping: 6/20 0.98568
stage 40/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:55 val_accuracy: 0.98552 early_stopping: 7/20 0.98568
stage 41/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:08 val_accuracy: 0.98525 early_stopping: 8/20 0.98568
stage 42/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:17 val_accuracy: 0.98563 early_stopping: 9/20 0.98568
stage 43/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:35:08 val_accuracy: 0.98507 early_stopping: 10/20 0.98568
stage 44/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:22 val_accuracy: 0.98513 early_stopping: 11/20 0.98568
stage 45/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:54 val_accuracy: 0.98502 early_stopping: 12/20 0.98568
stage 46/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:55 val_accuracy: 0.98499 early_stopping: 13/20 0.98568
stage 47/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:37 val_accuracy: 0.98489 early_stopping: 14/20 0.98568
stage 48/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:17 val_accuracy: 0.98521 early_stopping: 15/20 0.98568
stage 49/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:44 val_accuracy: 0.98497 early_stopping: 16/20 0.98568
stage 50/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:51 val_accuracy: 0.98511 early_stopping: 17/20 0.98568
stage 51/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:21 val_accuracy: 0.98438 early_stopping: 18/20 0.98568
stage 52/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:50 val_accuracy: 0.98581 early_stopping: 0/20 0.98581
stage 53/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:35:09 val_accuracy: 0.98514 early_stopping: 1/20 0.98581
stage 54/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:51 val_accuracy: 0.98528 early_stopping: 2/20 0.98581
stage 55/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:27 val_accuracy: 0.98537 early_stopping: 3/20 0.98581
stage 56/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:52 val_accuracy: 0.98514 early_stopping: 4/20 0.98581
stage 57/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:11 val_accuracy: 0.98447 early_stopping: 5/20 0.98581
stage 58/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:48 val_accuracy: 0.98503 early_stopping: 6/20 0.98581
stage 59/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:53 val_accuracy: 0.98478 early_stopping: 7/20 0.98581
stage 60/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:02 val_accuracy: 0.98477 early_stopping: 8/20 0.98581
stage 61/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:18 val_accuracy: 0.98538 early_stopping: 9/20 0.98581
stage 62/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:01 val_accuracy: 0.98488 early_stopping: 10/20 0.98581
stage 63/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:41 val_accuracy: 0.98533 early_stopping: 11/20 0.98581
stage 64/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:34:45 val_accuracy: 0.98488 early_stopping: 12/20 0.98581
stage 65/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:35:07 val_accuracy: 0.98522 early_stopping: 13/20 0.98581
stage 66/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:16 val_accuracy: 0.98546 early_stopping: 14/20 0.98581
stage 67/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:55 val_accuracy: 0.98497 early_stopping: 15/20 0.98581
stage 68/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:01 val_accuracy: 0.98494 early_stopping: 16/20 0.98581
stage 69/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:33:13 val_accuracy: 0.98489 early_stopping: 17/20 0.98581
stage 70/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:51 val_accuracy: 0.98577 early_stopping: 18/20 0.98581
stage 71/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:31:54 val_accuracy: 0.98463 early_stopping: 19/20 0.98581
stage 72/β ββββββββββββββββββββββββββββββββββββββββ 58692/58692 0:00:00 1:32:24 val_accuracy: 0.98452 early_stopping: 20/20 0.98581
Moving best model austriannewspapers_52.mlmodel (0.9858119487762451) to austriannewspapers_best.mlmodel
real 6813m24.688s
user 15739m27.931s
sys 26308m35.655s
(venv3.9) stweil@ocr-02:~/src/github/UB-Mannheim/AustrianNewspapers$ ls -lt
total 1173956
-rw-r--r-- 1 stweil stweil 16243476 May 25 14:54 austriannewspapers_best.mlmodel
-rw-r--r-- 1 stweil stweil 16244093 May 25 14:54 austriannewspapers_72.mlmodel
-rw-r--r-- 1 stweil stweil 16244062 May 25 13:22 austriannewspapers_71.mlmodel
[...]
-rw-r--r-- 1 stweil stweil 752 May 20 21:14 list.eval
-rw-r--r-- 1 stweil stweil 8274 May 20 21:14 list.train
drwxr-xr-x 1 stweil stweil 66 May 20 21:13 venv3.9
drwxr-xr-x 1 stweil stweil 18 May 20 21:12 gt
drwxr-xr-x 1 stweil stweil 1240 May 20 21:12 ValidationSet_ONB_Newseye_GT_M1+
drwxr-xr-x 1 stweil stweil 14152 May 20 21:12 TrainingSet_ONB_Newseye_GT_M1+
-rw-r--r-- 1 stweil stweil 2372 May 20 21:12 README.md
GPU Load 15...35 %, GPU Memory 2.8 GiB, time / epoch 1:25 h
# Prepare the training.
cd data
ls TrainingSet_ONB_Newseye_GT_M1+/GT-PAGE/*xml >list.train
ls ValidationSet_ONB_Newseye_GT_M1+/GT-PAGE/*xml >list.eval
(venv-3.9) stweil@ocr-02:~/src/github/UB-Mannheim/AustrianNewspapers/data$ time nice ketos train -f page -t list.train -e list.eval -o austriannewspapers -d cuda:0 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
scikit-learn version 1.2.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
[04/26/23 20:47:35] WARNING alphabet mismatch: chars in training set only: {'Ε', 'Ε', 'β‘', 'Β³', 'βΉ', 'Γ΄', 'Ε ', 'β', 'Γͺ', 'β', 'Γ»', 'β°', 'ΕΎ', '#', 'β', 'Γ±', 'βΈ«', 'β', 'βΆ', 'Γ«', 'β
', 'Γ²', 'βΈ', train.py:386
'Γ’', 'Γ', 'β', 'Λ’', 'β', 'β
', 'β³', 'β ', '"', 'β
', 'β', 'β€', 'β
', 'β―', 'Β±', 'Γ¦', 'Γ³', 'β', 'Γ', 'Ε', 'β
', 'β', 'β
', 'β ', 'Γ', 'ΒΉ', 'β
', 'β
', 'β', 'βΌ', 'β·', 'β
',
'β΄', 'β', 'β', 'β
', 'β', 'β', 'β', 'β²', '⬀', 'Β°', 'β', 'β΅'} (not included in accuracy test during training)
WARNING alphabet mismatch: chars in validation set only: {'Ε'} (not trained) train.py:390
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA RTX A5000') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
ββββββ³ββββββββββββ³βββββββββββββββββββββββββββ³βββββββββ³βββββββββββββββββββββββββββ³βββββββββββββββββββββββββββ
β β Name β Type β Params β In sizes β Out sizes β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β 0 β val_cer β CharErrorRate β 0 β ? β ? β
β 1 β val_wer β WordErrorRate β 0 β ? β ? β
β 2 β net β MultiParamSequential β 4.1 M β [[1, 1, 120, 400], '?'] β [[1, 187, 1, 50], '?'] β
β 3 β net.C_0 β ActConv2D β 1.3 K β [[1, 1, 120, 400], '?'] β [[1, 32, 120, 400], '?'] β
β 4 β net.Do_1 β Dropout β 0 β [[1, 32, 120, 400], '?'] β [[1, 32, 120, 400], '?'] β
β 5 β net.Mp_2 β MaxPool β 0 β [[1, 32, 120, 400], '?'] β [[1, 32, 60, 200], '?'] β
β 6 β net.C_3 β ActConv2D β 40.0 K β [[1, 32, 60, 200], '?'] β [[1, 32, 60, 200], '?'] β
β 7 β net.Do_4 β Dropout β 0 β [[1, 32, 60, 200], '?'] β [[1, 32, 60, 200], '?'] β
β 8 β net.Mp_5 β MaxPool β 0 β [[1, 32, 60, 200], '?'] β [[1, 32, 30, 100], '?'] β
β 9 β net.C_6 β ActConv2D β 55.4 K β [[1, 32, 30, 100], '?'] β [[1, 64, 30, 100], '?'] β
β 10 β net.Do_7 β Dropout β 0 β [[1, 64, 30, 100], '?'] β [[1, 64, 30, 100], '?'] β
β 11 β net.Mp_8 β MaxPool β 0 β [[1, 64, 30, 100], '?'] β [[1, 64, 15, 50], '?'] β
β 12 β net.C_9 β ActConv2D β 110 K β [[1, 64, 15, 50], '?'] β [[1, 64, 15, 50], '?'] β
β 13 β net.Do_10 β Dropout β 0 β [[1, 64, 15, 50], '?'] β [[1, 64, 15, 50], '?'] β
β 14 β net.S_11 β Reshape β 0 β [[1, 64, 15, 50], '?'] β [[1, 960, 1, 50], '?'] β
β 15 β net.L_12 β TransposedSummarizingRNN β 1.9 M β [[1, 960, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 16 β net.Do_13 β Dropout β 0 β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 17 β net.L_14 β TransposedSummarizingRNN β 963 K β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 18 β net.Do_15 β Dropout β 0 β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 19 β net.L_16 β TransposedSummarizingRNN β 963 K β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 20 β net.Do_17 β Dropout β 0 β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 21 β net.O_18 β LinSoftmax β 75.0 K β [[1, 400, 1, 50], '?'] β [[1, 187, 1, 50], '?'] β
ββββββ΄ββββββββββββ΄βββββββββββββββββββββββββββ΄βββββββββ΄βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββ
Trainable params: 4.1 M
Non-trainable params: 0
Total params: 4.1 M
Total estimated model params size (MB): 16
stage 0/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:07 β’ 0:00:00 10.54it/s val_accuracy: 0.977 val_word_accuracy: 0.883 early_stopping: 0/20 0.97716
stage 1/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:36 β’ 0:00:00 10.90it/s val_accuracy: 0.986 val_word_accuracy: 0.925 early_stopping: 0/20 0.98556
stage 2/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:22:47 β’ 0:00:00 11.03it/s val_accuracy: 0.988 val_word_accuracy: 0.934 early_stopping: 0/20 0.98769
stage 3/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:07 β’ 0:00:00 11.29it/s val_accuracy: 0.989 val_word_accuracy: 0.942 early_stopping: 0/20 0.98904
stage 4/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:11 β’ 0:00:00 11.24it/s val_accuracy: 0.99 val_word_accuracy: 0.945 early_stopping: 0/20 0.98967
stage 5/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:07 β’ 0:00:00 11.09it/s val_accuracy: 0.991 val_word_accuracy: 0.95 early_stopping: 0/20 0.99075
stage 6/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:34 β’ 0:00:00 10.94it/s val_accuracy: 0.991 val_word_accuracy: 0.949 early_stopping: 1/20 0.99075
stage 7/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:07 β’ 0:00:00 10.87it/s val_accuracy: 0.991 val_word_accuracy: 0.95 early_stopping: 0/20 0.99085
stage 8/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:08 β’ 0:00:00 11.27it/s val_accuracy: 0.991 val_word_accuracy: 0.953 early_stopping: 0/20 0.99123
stage 9/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:07 β’ 0:00:00 10.54it/s val_accuracy: 0.991 val_word_accuracy: 0.95 early_stopping: 1/20 0.99123
stage 10/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:54 β’ 0:00:00 10.69it/s val_accuracy: 0.992 val_word_accuracy: 0.953 early_stopping: 0/20 0.99153
stage 11/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:10 β’ 0:00:00 10.50it/s val_accuracy: 0.991 val_word_accuracy: 0.949 early_stopping: 1/20 0.99153
stage 12/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:01 β’ 0:00:00 10.85it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 0/20 0.99186
stage 13/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:22:50 β’ 0:00:00 10.96it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 0/20 0.99196
stage 14/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:33 β’ 0:00:00 11.27it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 0/20 0.99211
stage 15/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:08 β’ 0:00:00 10.95it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 1/20 0.99211
stage 16/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:38 β’ 0:00:00 10.85it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 2/20 0.99211
stage 17/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:43 β’ 0:00:00 11.50it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 3/20 0.99211
stage 18/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:36 β’ 0:00:00 10.88it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99212
stage 19/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:29 β’ 0:00:00 10.67it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 1/20 0.99212
stage 20/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:11 β’ 0:00:00 10.77it/s val_accuracy: 0.991 val_word_accuracy: 0.953 early_stopping: 2/20 0.99212
stage 21/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:22:50 β’ 0:00:00 11.19it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 3/20 0.99212
stage 22/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:31 β’ 0:00:00 11.10it/s val_accuracy: 0.992 val_word_accuracy: 0.954 early_stopping: 4/20 0.99212
stage 23/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:48 β’ 0:00:00 10.48it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 5/20 0.99212
stage 24/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:35 β’ 0:00:00 10.66it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 6/20 0.99212
stage 25/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:24 β’ 0:00:00 11.17it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 7/20 0.99212
stage 26/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:17 β’ 0:00:00 11.05it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99222
stage 27/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:35 β’ 0:00:00 10.88it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99231
stage 28/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:48 β’ 0:00:00 10.80it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99234
stage 29/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:05 β’ 0:00:00 10.91it/s val_accuracy: 0.993 val_word_accuracy: 0.959 early_stopping: 0/20 0.99251
stage 30/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:30 β’ 0:00:00 10.88it/s val_accuracy: 0.991 val_word_accuracy: 0.95 early_stopping: 1/20 0.99251
stage 31/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:32 β’ 0:00:00 11.01it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 3/20 0.99251
stage 33/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:46 β’ 0:00:00 10.72it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 4/20 0.99251
stage 34/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:01 β’ 0:00:00 10.81it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 5/20 0.99251
stage 35/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:07 β’ 0:00:00 10.94it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 6/20 0.99251
stage 36/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:12 β’ 0:00:00 11.16it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 7/20 0.99251
stage 37/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:28 β’ 0:00:00 10.81it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 8/20 0.99251
stage 38/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:16 β’ 0:00:00 11.09it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 8/20 0.99251
Validation βββββββββββββββββΈβββββββββββββββββββββββ 2088/5016 0:03:11 β’ 0:04:37 10.58it/s early_stopping: 8/20 0.99251
stage 8/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:08 β’ 0:00:00 11.27it/s val_accuracy: 0.991 val_word_accuracy: 0.953 early_stopping: 0/20 0.99123
stage 9/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:07 β’ 0:00:00 10.54it/s val_accuracy: 0.991 val_word_accuracy: 0.95 early_stopping: 1/20 0.99123
stage 10/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:54 β’ 0:00:00 10.69it/s val_accuracy: 0.992 val_word_accuracy: 0.953 early_stopping: 0/20 0.99153
stage 11/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:10 β’ 0:00:00 10.50it/s val_accuracy: 0.991 val_word_accuracy: 0.949 early_stopping: 1/20 0.99153
stage 12/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:01 β’ 0:00:00 10.85it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 0/20 0.99186
stage 13/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:22:50 β’ 0:00:00 10.96it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 0/20 0.99196
stage 14/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:33 β’ 0:00:00 11.27it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 0/20 0.99211
stage 15/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:08 β’ 0:00:00 10.95it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 1/20 0.99211
stage 16/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:38 β’ 0:00:00 10.85it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 2/20 0.99211
stage 17/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:43 β’ 0:00:00 11.50it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 3/20 0.99211
stage 18/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:36 β’ 0:00:00 10.88it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99212
stage 19/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:29 β’ 0:00:00 10.67it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 1/20 0.99212
stage 20/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:11 β’ 0:00:00 10.77it/s val_accuracy: 0.991 val_word_accuracy: 0.953 early_stopping: 2/20 0.99212
stage 21/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:22:50 β’ 0:00:00 11.19it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 3/20 0.99212
stage 22/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:31 β’ 0:00:00 11.10it/s val_accuracy: 0.992 val_word_accuracy: 0.954 early_stopping: 4/20 0.99212
stage 23/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:48 β’ 0:00:00 10.48it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 5/20 0.99212
stage 24/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:35 β’ 0:00:00 10.66it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 6/20 0.99212
stage 25/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:24 β’ 0:00:00 11.17it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 7/20 0.99212
stage 26/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:17 β’ 0:00:00 11.05it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99222
stage 27/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:35 β’ 0:00:00 10.88it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99231
stage 28/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:48 β’ 0:00:00 10.80it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/20 0.99234
stage 29/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:05 β’ 0:00:00 10.91it/s val_accuracy: 0.993 val_word_accuracy: 0.959 early_stopping: 0/20 0.99251
stage 30/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:30 β’ 0:00:00 10.88it/s val_accuracy: 0.991 val_word_accuracy: 0.95 early_stopping: 1/20 0.99251
stage 31/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:32 β’ 0:00:00 11.01it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 2/20 0.99251
stage 32/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:30 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 3/20 0.99251
stage 33/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:46 β’ 0:00:00 10.72it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 4/20 0.99251
stage 34/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:25:01 β’ 0:00:00 10.81it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 5/20 0.99251
stage 35/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:07 β’ 0:00:00 10.94it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 6/20 0.99251
stage 36/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:12 β’ 0:00:00 11.16it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 7/20 0.99251
stage 37/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:28 β’ 0:00:00 10.81it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 8/20 0.99251
stage 38/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:16 β’ 0:00:00 11.09it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 8/20 0.99251
stage 38/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:16 β’ 0:00:00 11.09it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 9/20 0.99251
stage 39/β ββββββββββββββββββββββββββββββββββββββββ 285/54824 0:00:25 β’ 1:22:19 11.04it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 9/20 0.99251
stage 39/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:12 β’ 0:00:00 11.19it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 10/20 0.99251
stage 40/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:32 β’ 0:00:00 11.03it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 11/20 0.99251
stage 41/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:20 β’ 0:00:00 11.11it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 12/20 0.99251
stage 42/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:45 β’ 0:00:00 10.41it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 13/20 0.99251
stage 43/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:24:08 β’ 0:00:00 11.05it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 14/20 0.99251
stage 44/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:31 β’ 0:00:00 10.99it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 15/20 0.99251
stage 45/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:38 β’ 0:00:00 10.97it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 16/20 0.99251
stage 46/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:28 β’ 0:00:00 10.68it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 17/20 0.99251
stage 47/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:18 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 17/20 0.99251
Validation βββββββββββββββββββββββββββββββΊβββββββββ 3816/5016 0:05:47 β’ 0:01:57 10.26it/s early_stopping: 17/20 0.99251
stage 47/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:18 β’ 0:00:00 10.78it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 18/20 0.99251
stage 48/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:18 β’ 0:00:00 10.92it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 19/20 0.99251
stage 49/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:23:26 β’ 0:00:00 10.95it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 20/20 0.99251
Moving best model austriannewspapers_29.mlmodel (0.9925086498260498) to austriannewspapers_best.mlmodel
real 4554m17,818s
user 11489m52,859s
sys 18042m2,729s
ocr-01, decomposed GT
(venv3.9) stweil@ocr-01:~/src/github/UB-Mannheim/AustrianNewspapers/data$ time nice ketos train -f page -t list.train -e list.eval -o austriannewspapers -d cuda:0 --lag 10 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
scikit-learn version 1.2.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
[04/29/23 20:44:52] WARNING alphabet mismatch: chars in training set only: {'β', 'β·', '"', 'β', 'β‘', 'β', 'β', 'βΈ«', 'β΄', 'β΅', 'β', 'β', 'β', '#', 'ΒΉ', 'β', 'β―', 'β ', 'Μ', 'βΉ', 'β', 'Μ', 'βΌ', 'β
', 'Β³', train.py:387
'β
', 'β', 'β
', 'Γ¦', '⬀', 'βΆ', 'β', 'Λ’', 'β', 'β€', 'β', 'β', 'Β°', 'β³', 'β°', 'βΈ', 'Μ', 'β ', 'Β±', 'β²', 'β'} (not included in accuracy test during training)
WARNING alphabet mismatch: chars in validation set only: {'Μ'} (not trained) train.py:391
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
ββββββ³ββββββββββββ³βββββββββββββββββββββββββββ³βββββββββ³βββββββββββββββββββββββββββ³βββββββββββββββββββββββββββ
β β Name β Type β Params β In sizes β Out sizes β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β 0 β val_cer β CharErrorRate β 0 β ? β ? β
β 1 β val_wer β WordErrorRate β 0 β ? β ? β
β 2 β net β MultiParamSequential β 4.1 M β [[1, 1, 120, 400], '?'] β [[1, 152, 1, 50], '?'] β
β 3 β net.C_0 β ActConv2D β 1.3 K β [[1, 1, 120, 400], '?'] β [[1, 32, 120, 400], '?'] β
β 4 β net.Do_1 β Dropout β 0 β [[1, 32, 120, 400], '?'] β [[1, 32, 120, 400], '?'] β
β 5 β net.Mp_2 β MaxPool β 0 β [[1, 32, 120, 400], '?'] β [[1, 32, 60, 200], '?'] β
β 6 β net.C_3 β ActConv2D β 40.0 K β [[1, 32, 60, 200], '?'] β [[1, 32, 60, 200], '?'] β
β 7 β net.Do_4 β Dropout β 0 β [[1, 32, 60, 200], '?'] β [[1, 32, 60, 200], '?'] β
β 8 β net.Mp_5 β MaxPool β 0 β [[1, 32, 60, 200], '?'] β [[1, 32, 30, 100], '?'] β
β 9 β net.C_6 β ActConv2D β 55.4 K β [[1, 32, 30, 100], '?'] β [[1, 64, 30, 100], '?'] β
β 10 β net.Do_7 β Dropout β 0 β [[1, 64, 30, 100], '?'] β [[1, 64, 30, 100], '?'] β
β 11 β net.Mp_8 β MaxPool β 0 β [[1, 64, 30, 100], '?'] β [[1, 64, 15, 50], '?'] β
β 12 β net.C_9 β ActConv2D β 110 K β [[1, 64, 15, 50], '?'] β [[1, 64, 15, 50], '?'] β
β 13 β net.Do_10 β Dropout β 0 β [[1, 64, 15, 50], '?'] β [[1, 64, 15, 50], '?'] β
β 14 β net.S_11 β Reshape β 0 β [[1, 64, 15, 50], '?'] β [[1, 960, 1, 50], '?'] β
β 15 β net.L_12 β TransposedSummarizingRNN β 1.9 M β [[1, 960, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 16 β net.Do_13 β Dropout β 0 β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 17 β net.L_14 β TransposedSummarizingRNN β 963 K β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 18 β net.Do_15 β Dropout β 0 β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 19 β net.L_16 β TransposedSummarizingRNN β 963 K β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 20 β net.Do_17 β Dropout β 0 β [[1, 400, 1, 50], '?'] β [[1, 400, 1, 50], '?'] β
β 21 β net.O_18 β LinSoftmax β 61.0 K β [[1, 400, 1, 50], '?'] β [[1, 152, 1, 50], '?'] β
ββββββ΄ββββββββββββ΄βββββββββββββββββββββββββββ΄βββββββββ΄βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββ
Trainable params: 4.1 M Non-trainable params: 0
Total params: 4.1 M Total estimated model params size (MB): 16
stage 0/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:38:07 β’ 0:00:00 9.71it/s val_accuracy: 0.981 val_word_accuracy: 0.901 early_stopping: 0/10 0.98058
stage 1/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:39:17 β’ 0:00:00 9.45it/s val_accuracy: 0.987 val_word_accuracy: 0.93 early_stopping: 0/10 0.98672
stage 2/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:38:26 β’ 0:00:00 9.37it/s val_accuracy: 0.988 val_word_accuracy: 0.935 early_stopping: 0/10 0.98786
stage 3/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:35 β’ 0:00:00 9.84it/s val_accuracy: 0.99 val_word_accuracy: 0.945 early_stopping: 0/10 0.98961
stage 4/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:29 β’ 0:00:00 9.64it/s val_accuracy: 0.99 val_word_accuracy: 0.945 early_stopping: 0/10 0.98998
stage 5/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:36 β’ 0:00:00 9.74it/s val_accuracy: 0.99 val_word_accuracy: 0.946 early_stopping: 0/10 0.99007
stage 6/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:40 β’ 0:00:00 9.34it/s val_accuracy: 0.99 val_word_accuracy: 0.946 early_stopping: 0/10 0.99007
Validation ββββββββββββββββββββββββββββββββββββΈββββ 4455/5016 0:07:39 β’ 0:00:42 13.42it/s early_stopping: 0/10 0.99007
stage 6/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:40 β’ 0:00:00 9.34it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 0/10 0.99119
stage 6/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:40 β’ 0:00:00 9.34it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 0/10 0.99119
stage 7/β ββββββββββββββββββββββββββΈββββββββββββββ 35158/54824 1:01:37 β’ 0:33:24 9.81it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 0/10 0.99119
stage 7/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:58 β’ 0:00:00 9.30it/s val_accuracy: 0.991 val_word_accuracy: 0.953 early_stopping: 0/10 0.99141
stage 8/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:55 β’ 0:00:00 9.30it/s val_accuracy: 0.992 val_word_accuracy: 0.953 early_stopping: 0/10 0.99152
stage 9/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:56 β’ 0:00:00 9.53it/s val_accuracy: 0.991 val_word_accuracy: 0.954 early_stopping: 1/10 0.99152
stage 10/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:57 β’ 0:00:00 9.32it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 0/10 0.99179
stage 11/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:44 β’ 0:00:00 9.38it/s val_accuracy: 0.992 val_word_accuracy: 0.954 early_stopping: 1/10 0.99179
stage 12/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:44 β’ 0:00:00 9.64it/s val_accuracy: 0.991 val_word_accuracy: 0.954 early_stopping: 2/10 0.99179
stage 13/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:38 β’ 0:00:00 9.27it/s val_accuracy: 0.992 val_word_accuracy: 0.954 early_stopping: 3/10 0.99179
stage 14/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:36:05 β’ 0:00:00 9.44it/s val_accuracy: 0.992 val_word_accuracy: 0.955 early_stopping: 4/10 0.99179
stage 15/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:37:33 β’ 0:00:00 9.56it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 0/10 0.99210
stage 16/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:37:31 β’ 0:00:00 9.66it/s val_accuracy: 0.992 val_word_accuracy: 0.953 early_stopping: 1/10 0.99210
stage 17/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:36:24 β’ 0:00:00 9.34it/s val_accuracy: 0.992 val_word_accuracy: 0.958 early_stopping: 0/10 0.99245
stage 18/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:37:13 β’ 0:00:00 9.51it/s val_accuracy: 0.988 val_word_accuracy: 0.941 early_stopping: 1/10 0.99245
stage 19/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:37:30 β’ 0:00:00 9.50it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 2/10 0.99245
stage 20/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:42 β’ 0:00:00 9.46it/s val_accuracy: 0.991 val_word_accuracy: 0.952 early_stopping: 3/10 0.99245
stage 21/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:37:08 β’ 0:00:00 9.26it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 4/10 0.99245
stage 22/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:38:57 β’ 0:00:00 9.55it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 5/10 0.99245
stage 23/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:36:53 β’ 0:00:00 9.41it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 6/10 0.99245
stage 24/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:36:03 β’ 0:00:00 9.63it/s val_accuracy: 0.992 val_word_accuracy: 0.956 early_stopping: 7/10 0.99245
stage 25/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:15 β’ 0:00:00 9.62it/s val_accuracy: 0.992 val_word_accuracy: 0.954 early_stopping: 8/10 0.99245
stage 26/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:37:13 β’ 0:00:00 9.29it/s val_accuracy: 0.991 val_word_accuracy: 0.953 early_stopping: 9/10 0.99245
stage 27/β ββββββββββββββββββββββββββββββββββββββββ 54824/54824 1:35:23 β’ 0:00:00 9.64it/s val_accuracy: 0.992 val_word_accuracy: 0.957 early_stopping: 10/10 0.99245
Moving best model austriannewspapers_17.mlmodel (0.992448091506958) to austriannewspapers_best.mlmodel
real 2939m36.702s
user 7959m30.528s
sys 13192m38.440s