Training - UB-Mannheim/kraken GitHub Wiki
Training
Examples of trainings for kraken
This sequence of examples was focused on getting a good kraken model for historic prints, especially for Fraktur script, but also with support for historic and modern Antiqua fonts.
- https://github.com/UB-Mannheim/AustrianNewspapers/wiki/Training-with-Kraken
- https://github.com/UB-Mannheim/digi-gt/wiki/Training#training-mit-kraken
- https://github.com/UB-Mannheim/reichsanzeiger-gt/wiki/Training-with-Kraken
- https://github.com/UB-Mannheim/digitue-gt/wiki/Training-with-Kraken
frak2021
The training used the same data sets for training and evaluation as a previous training which was done with Tesseract.
It was done the latest kraken code plus additional fixes for a division by null and multithreading.
nohup time ketos train -t list.train -e list.eval -o frak2021_gpu -d cuda:0 | tee -a frak2021_gpu.log
nohup: Eingabe wird ignoriert und leite Standardfehlerausgabe auf Standardausgabe umgeleitet
Building training set
Building validation set
[52.2009] alphabet mismatch: chars in training set only: {'~', 'แฝ', 'ล', 'ร', 'ฯ', 'โ', 'ร', 'แฟณ', 'รฏ', 'แธก', 'ฯ', 'แผ', 'ฮผ', 'ฯ', 'โ', 'แผธ', 'ฯ', 'ล', 'แผฐ', 'ฮพ', 'อ', 'โ', 'โ
', 'ฤ', 'ฯ', 'ฯ', 'แฝ', 'ฮฆ', 'ฮ', '_', 'ฯฐ', 'ั', 'แผ', 'ฮค', 'แด', 'ฮบ', 'ร', 'ฮ', 'ร', 'โฒ', 'แผ', 'ร', 'ร', 'แฝฐ', '+', 'ฯ', 'โฃ', 'ลฝ', 'ฮท', 'โฒ', 'ฮ', 'ฮฏ', 'โ
', 'ลพ', 'ล ', 'โ', 'แฝ', 'ฯ', 'ยฑ', 'ฮฌ', 'ฯ', 'โ', 'รฎ', 'โ', 'โ', 'ฮธ', 'โ
', 'โณ', 'ร', 'ั', 'แฟ', 'ฯ', 'โ', 'ฯฑ', 'โค', 'แฝธ', 'โ', 'โ
', 'แผ', 'ฮด', 'แฝด', 'แผ', 'ษ', 'ฮถ', 'แพฝ', 'แฝถ', 'รฌ', 'ส', 'โ
', 'โ', 'โ', 'โ', 'ฮฎ'} (not included in accuracy test during training)
[52.2010] alphabet mismatch: chars in validation set only: {'ฯ'} (not trained)
Initializing model โ
stage 1/โ
Accuracy report (1) 0.9612 1625450 63133
stage 2/โ
Accuracy report (2) 0.9604 1625450 64311
stage 3/โ
Accuracy report (3) 0.9578 1625450 68579
stage 4/โ
Accuracy report (4) 0.9519 1625450 78227
stage 5/โ
Accuracy report (5) 0.9428 1625450 92982
stage 6/โ
Accuracy report (6) 0.9421 1625450 94036
Moving best model frak2021_gpu_1.mlmodel (0.9611597061157227) to frak2021_gpu_best.mlmodel
154547.59user 1537.72system 29:15:33elapsed 148%CPU (0avgtext+0avgdata 5916304maxresident)k
385328inputs+453328outputs (68102177major+26511973minor)pagefaults 0swaps
Juristische Konsilien Tรผbingen
Get the data
The information here is no longer valid.
mkdir Juristische_Konsilien_Tuebingen
cd Juristische_Konsilien_Tuebingen
mkdir Training
cd Training
# Get image files.
wget --content-disposition https://files.transkribus.eu/Get?id=KWPGLJROIYFMCHXUKHROTBZI https://files.transkribus.eu/Get?id=FFZEEILUCMKWBMTOJNKXOQVO https://files.transkribus.eu/Get?id=DHTNZFGEQZVNFYSNNXSUGRZW https://files.transkribus.eu/Get?id=GLUHQPHRJZGIORAKNKSSHJHN https://files.transkribus.eu/Get?id=AXGGUKSGAASTMJLXHKLHTCMI https://files.transkribus.eu/Get?id=ZWNHITSTVYAYVRMXSACMYZSS https://files.transkribus.eu/Get?id=SJOIKRULJGZRUGOHANSBQCES https://files.transkribus.eu/Get?id=TXUOIQXJUFRBXPZXRIQKRGID https://files.transkribus.eu/Get?id=FBFDPQSANWKZHMEFPCAEBYGC https://files.transkribus.eu/Get?id=JJYAAMDIJMVXEMDQNQLFSNHP https://files.transkribus.eu/Get?id=CFHAJXQXKFTYWEMOXGMHIFTK https://files.transkribus.eu/Get?id=YJNIOPLPTVNXGLVKAYDWSZTJ https://files.transkribus.eu/Get?id=KSSNVVQMKCIPCOJNVRQIOIIH https://files.transkribus.eu/Get?id=OGZBGTHHLDZQRUFVQBDMDDPA https://files.transkribus.eu/Get?id=KYPDSMEVHCRVHUXKYNOXDTPT https://files.transkribus.eu/Get?id=UJUHLJMVEUIXOOVDPIQKPXFL https://files.transkribus.eu/Get?id=ZTTKWCFYHHJXPCJHVLGCFQCM https://files.transkribus.eu/Get?id=FOVUVMNPDJDZLKWIDYGJPLUX https://files.transkribus.eu/Get?id=DXRPIBLIMBBMIXVDXCSMOLDM https://files.transkribus.eu/Get?id=LZPWBKIXSJJIFAWPGUZSPVXT https://files.transkribus.eu/Get?id=HOHGKGGHOSQUUTAVACVQYWAL https://files.transkribus.eu/Get?id=GPQZHFWYWLZEXJCDTJKAALEE https://files.transkribus.eu/Get?id=TAAKRFOWGFBKCEFSHNQIVOIK https://files.transkribus.eu/Get?id=RSAIQAPCKZZTNAKLEHTWBPGM https://files.transkribus.eu/Get?id=NZGMAVAGNRHFMZNNUIJDTRLU https://files.transkribus.eu/Get?id=PCNIVHAJHFRBOYBPQERJTOED https://files.transkribus.eu/Get?id=OJKITNXIXTYUQPCZKYHQYFSD https://files.transkribus.eu/Get?id=SZJTDCPESLWRIHCZXMBEENTJ https://files.transkribus.eu/Get?id=LTREBTDWWJRERTPPUBONYEEH https://files.transkribus.eu/Get?id=KLINCPGDKEXBUQNBUDGJUOPY https://files.transkribus.eu/Get?id=ZGQKNMDFPAZNRBMMKSOZZHKA https://files.transkribus.eu/Get?id=HJFOROVZUNTZMHRZJJOOKJAM https://files.transkribus.eu/Get?id=JAOKFQRGZUHUSDRJUWEGEGUE https://files.transkribus.eu/Get?id=ULKQLRYWHGTQMGGDRWWVKNOG https://files.transkribus.eu/Get?id=FXHKSRQIETJMKXBQZHLEHHZZ https://files.transkribus.eu/Get?id=RQSXJYKCEUBGMPTEHSCBVAAT https://files.transkribus.eu/Get?id=LHMCRHNCINSMVCINCEKNACCG https://files.transkribus.eu/Get?id=LFGZRQTADQHPYNUJPNHFOLMR https://files.transkribus.eu/Get?id=XVTTWUSOAQFYRSZQVLQMMNYB https://files.transkribus.eu/Get?id=MQROMFPZURWSUWVWJAFNJWMS https://files.transkribus.eu/Get?id=XOIOSHOVCUSCCVHNGLOWBUHA https://files.transkribus.eu/Get?id=MQQGCETBKJUWDLIQYUKPRJRU https://files.transkribus.eu/Get?id=HOBEDMUTPAPQPGCVBFSSSUJS https://files.transkribus.eu/Get?id=RYRIDAQAQCWEUCFZSOJMIEVT https://files.transkribus.eu/Get?id=FKUXUMUMEVVMYEPKVXUTCKHY https://files.transkribus.eu/Get?id=ROOUZCYQSEZSDEBSHEMGPQCB https://files.transkribus.eu/Get?id=HYFOIAPWWUYVLBKKNCWWVQDR https://files.transkribus.eu/Get?id=MBLXSOIAFBZPRGCBEAWLSXIQ https://files.transkribus.eu/Get?id=JPTFEKQFLOIJABAHZPJGWTIX https://files.transkribus.eu/Get?id=GVOZSINXFOQINGIHOFLHZEXW https://files.transkribus.eu/Get?id=UIUAZQUCPLZMULJHNUGEYUKR https://files.transkribus.eu/Get?id=AJJVYMFQTWWIWJFCUKMJKJQP https://files.transkribus.eu/Get?id=GIJZOLKNHKZXBTBWPJJJLYYR https://files.transkribus.eu/Get?id=KRXUUEYRYVHLMMUFLKCGUQQH https://files.transkribus.eu/Get?id=RKUEWSQAKAWVGCJMBNOSIVNR https://files.transkribus.eu/Get?id=MNOPSPMISKMPTXUIJUSOEOCW https://files.transkribus.eu/Get?id=BFYUPHCCJDCUAZWITORDMLGX https://files.transkribus.eu/Get?id=APIACVAWACKQMPYJILFINMZT https://files.transkribus.eu/Get?id=TVBDRWEFCSENIIYFTLRYPEAA https://files.transkribus.eu/Get?id=GJYRHFRDWSHRLJVIGNKWWBZN https://files.transkribus.eu/Get?id=BQDKZIVQUJEXYYBCNVTBLHZM https://files.transkribus.eu/Get?id=DOIIRJFDXVHNAXFMUVFBPPYE https://files.transkribus.eu/Get?id=CSELZDAMABXKPWZNCSOHCBAF https://files.transkribus.eu/Get?id=SGFNBPQGXZMZAKGUSWXWOPVR https://files.transkribus.eu/Get?id=XQQIAQLQRMKMYCVHCLWHUYFQ https://files.transkribus.eu/Get?id=ROMIWFNFIBTGGCFLIZJJFWKU https://files.transkribus.eu/Get?id=KTUVRXMZBRHUYFVDAULYHKBC https://files.transkribus.eu/Get?id=AQHXCBAWQVKHUVUHZGDZPZUT https://files.transkribus.eu/Get?id=RXKIXYMKMLQBTJGXFYGCSGDY https://files.transkribus.eu/Get?id=UBEIKSGVTBMHIJPQOEGZSGWD https://files.transkribus.eu/Get?id=RWMVMWDCPNFBTQFGWQOAAVQQ https://files.transkribus.eu/Get?id=NYDHDAZGYHUDGXENOAASLFYF https://files.transkribus.eu/Get?id=SMVJNOMKZTUVAXKVDAJXSAZS https://files.transkribus.eu/Get?id=VIHRDXWGGOATSMCPZJJXAAXJ https://files.transkribus.eu/Get?id=YWFJNUVWHSZGMBDEZTJMCGFM https://files.transkribus.eu/Get?id=OXHBEFWYPNGMTTARWUPYSYUU https://files.transkribus.eu/Get?id=NDCENLOGOVSMHFPOTPOUCQKG https://files.transkribus.eu/Get?id=TIGUOIFRBCUMNOFZYARPGGRS https://files.transkribus.eu/Get?id=NKDMDKVMRTZWTKTRMCYDHFLW https://files.transkribus.eu/Get?id=IDIHIPZTDOXTOQCAWCBQMNOQ https://files.transkribus.eu/Get?id=IQOTZUSGWOKZEPBJBSYYJGTM https://files.transkribus.eu/Get?id=FRWNLSWXMYIPGQJGVPLWPIXT https://files.transkribus.eu/Get?id=QNXDOMQULVGUWUGHIKQDPCOB https://files.transkribus.eu/Get?id=EYISIDNNZSWKOQOYMQUTXVSK https://files.transkribus.eu/Get?id=RGFHPRMBPRYNEFLVEJTCOILG https://files.transkribus.eu/Get?id=FBZILEXIIRGVPPJLOYYSGHCD https://files.transkribus.eu/Get?id=KOIUIIXSXVJDFERSFIXMSZUL https://files.transkribus.eu/Get?id=OYLYXYLBLJYUJYOPLEFDHKLH https://files.transkribus.eu/Get?id=KJVSKBNNSJOQFGSRXVMRZYQR https://files.transkribus.eu/Get?id=UHAHGFFAMRNSTNQGMOBNQTUR https://files.transkribus.eu/Get?id=ADDQMOOXTWMOXGEEYMANWNJZ https://files.transkribus.eu/Get?id=HESNYZGFTYGUDFGZANLQLMOF https://files.transkribus.eu/Get?id=NQCXKFIDTGDACYHFUJZKJQDS https://files.transkribus.eu/Get?id=JZUMSCTEYEHMAOXDAXZNXGUX https://files.transkribus.eu/Get?id=IBERFVXKQCGEAHQNKPNCSNMA https://files.transkribus.eu/Get?id=FGLWDUKYAMWQIHEHKKQOOLMW https://files.transkribus.eu/Get?id=BCUVIEVSNFOUAYXSYCHNPXVP https://files.transkribus.eu/Get?id=XGPDURDECYWHYWZLJHDMIGLU https://files.transkribus.eu/Get?id=YSSYTELQQLMAHAVXQOGPUTUG https://files.transkribus.eu/Get?id=JOYKOTRPSTUFZXBORFKRCZJC https://files.transkribus.eu/Get?id=USOQSWUKNUEATWIZHMUYLDAF https://files.transkribus.eu/Get?id=VKIXEWUCBSZSKBBGIBOVUSUG https://files.transkribus.eu/Get?id=MEYLSJNTPSHUCXXUAJKCZPKA https://files.transkribus.eu/Get?id=XJDDZZAHHOSJGEPLXOBBRDOA https://files.transkribus.eu/Get?id=MYHOGTLJMXZUVRLJOSCUWWFC https://files.transkribus.eu/Get?id=BLASBUUDLXUNZMUIEVJUEJKJ https://files.transkribus.eu/Get?id=WGPNKBRBCKXVXKFONPETTISL https://files.transkribus.eu/Get?id=QMZIHCDKWOSFAPQVPENYRPHJ https://files.transkribus.eu/Get?id=KONWXWVKBGRGADHIGCFGFOXO https://files.transkribus.eu/Get?id=XXQAPZNSVNXIPGRGVHPTABFI https://files.transkribus.eu/Get?id=IPQPOXCWANMWHGBRMRKIYZLS https://files.transkribus.eu/Get?id=VIMEFFPNDFRWXEJQNXODRBEO https://files.transkribus.eu/Get?id=ZAABPSRTGFETETNGYMGLZCOL https://files.transkribus.eu/Get?id=VLSHOPUQJNHGWWSZKKXQAAHK https://files.transkribus.eu/Get?id=HZIGXWAUPNIABHHIBQHBTGEC https://files.transkribus.eu/Get?id=WDHNPKASTGLNHDIVJKSRVFMX https://files.transkribus.eu/Get?id=ZQWHSMWYZFDJUUDMGEYJYDDY https://files.transkribus.eu/Get?id=ZECCNUCTLIHMSLJPJVGAWUXP https://files.transkribus.eu/Get?id=XKKHGWQTAHWAULBAZBLBQCBQ https://files.transkribus.eu/Get?id=BDFTIFSQHNPGIQHLUSQBNZGL https://files.transkribus.eu/Get?id=NMFPJUPMQPKXBBLCBLNVDUQA https://files.transkribus.eu/Get?id=ZANEBGCYUOUQLYXHKOVSPMER https://files.transkribus.eu/Get?id=EGEUWBWWBYWLKJISSMKERPRZ https://files.transkribus.eu/Get?id=YFACZWEGHUMIJOSGFDIFIIXI https://files.transkribus.eu/Get?id=WXWUUGZKGYIKYFJVUPXLXNLN https://files.transkribus.eu/Get?id=BHERQQEMBDPKHNQCTTIABRAN https://files.transkribus.eu/Get?id=DKFQEVFTRHFUYTNWUWKGMDCC https://files.transkribus.eu/Get?id=MHUFOPWVDKPNENGOTPHMRBPG https://files.transkribus.eu/Get?id=PMKYLRJEIQBIIFQVGAAKBAYE https://files.transkribus.eu/Get?id=BTHWVSEFZFEQVWQXNLLVMQQX https://files.transkribus.eu/Get?id=YWYAOOYGHXSNXEMOWXFVFBBV https://files.transkribus.eu/Get?id=ZARBXUTLMISJJORKXYCCLKLJ https://files.transkribus.eu/Get?id=VIIJFLMYIRGLEPGPPAEUOHEG https://files.transkribus.eu/Get?id=BJAVSUFRATHIZMZJBFFVZLWY https://files.transkribus.eu/Get?id=XAFFBZVXTIOMURIZJANEDCRL https://files.transkribus.eu/Get?id=EADIASJDESBUIGBHBBRVBENC https://files.transkribus.eu/Get?id=UHXRNRWZURNNIITNZLBHIKAT https://files.transkribus.eu/Get?id=LYMSCWGIRTABXAXYSWMKLXFF https://files.transkribus.eu/Get?id=RVHBELCYHNQXRVOVSFHFPHUF https://files.transkribus.eu/Get?id=WYPAPFXUXDIYSHGCVUHPCDOE https://files.transkribus.eu/Get?id=RAAORHEZEHXFRPDCFVOUDSNE https://files.transkribus.eu/Get?id=AUMCSARBRKNJLWQGOEFAVEVN https://files.transkribus.eu/Get?id=LHPLORVBMFYIQXIFUOOQNWLF https://files.transkribus.eu/Get?id=YECVIISDTADFSAVYVVRULJED https://files.transkribus.eu/Get?id=BFQTOOGEOSYCFSTIJFYASZKN https://files.transkribus.eu/Get?id=PJXUZAFEEIUWRLDOENGGGYQM https://files.transkribus.eu/Get?id=YOHJAMQIAVSFHWRUESLIHEJH https://files.transkribus.eu/Get?id=EJXPQXSUQUIUSHKMECISPLRE https://files.transkribus.eu/Get?id=ETRCBMFJJFFSCBWGCWMKVYPU https://files.transkribus.eu/Get?id=RSYKJGUJTWKLWHUVWCZGVHBK https://files.transkribus.eu/Get?id=LHMYRDDJSVVHXCFZAPPIUYWS https://files.transkribus.eu/Get?id=JUQMUFIOQAYNHJSATWZVAVJK https://files.transkribus.eu/Get?id=WJLWJKXFVIOXLXEELWFAXEWS https://files.transkribus.eu/Get?id=XVFBFYOABAYUUBGEETZFNDUU https://files.transkribus.eu/Get?id=GSSITABWCVXAEYCPEGGAGWHO https://files.transkribus.eu/Get?id=HHQPEJLPUUSTOJVMGYHEMCWO https://files.transkribus.eu/Get?id=ETKNGULSXICMTXMSUNGUHAGR https://files.transkribus.eu/Get?id=EVVBDQATMBBOWZQVWWBGWFTE https://files.transkribus.eu/Get?id=JQNZVEEWHOTGPSGQWBJTYDXZ https://files.transkribus.eu/Get?id=HDJMNCKXBJTYRBANCDDNBOTV https://files.transkribus.eu/Get?id=FXWRRMVBGADVUBCLXQTUYPAT https://files.transkribus.eu/Get?id=YHIBTJOSJFQPMEFCMUASZYRR https://files.transkribus.eu/Get?id=WUALYWRDKBXDJKNGZGXEYHKQ https://files.transkribus.eu/Get?id=IENQXLHPZTOZJEENEIKRAKNU https://files.transkribus.eu/Get?id=ABWEDGWBHNRUHCDAZQBCHDHR https://files.transkribus.eu/Get?id=HOHZGRKUZEEHTQFOZIUVWRKG https://files.transkribus.eu/Get?id=TYEHTOEKTZMMCEFACTOKAUFI https://files.transkribus.eu/Get?id=DWSKPMUEJAIOJJYYGIRBIGCQ https://files.transkribus.eu/Get?id=JOUYTXSVTWDUMSRJHPBXFGKX https://files.transkribus.eu/Get?id=FGDLEJDZPQNYAMEAQXIQLDNK https://files.transkribus.eu/Get?id=JUXAJMEYQARQLPURIWTWGBBP https://files.transkribus.eu/Get?id=KRNJXKZJCTBMJJDLHHMMSINE https://files.transkribus.eu/Get?id=YULOHZIMFHEOJKWDHCFOLBUL https://files.transkribus.eu/Get?id=OFTLVWCMKVUPUZZJCRLXEIYS https://files.transkribus.eu/Get?id=YHRJAICVWUIQGVCSIPBMKEKH https://files.transkribus.eu/Get?id=JIYRQKQXDYYQJOFINPLTOQSZ https://files.transkribus.eu/Get?id=SSASFJCYPAANTCROPGETZUPP https://files.transkribus.eu/Get?id=QEXKWYLPSRKUWBCIGQNHHPTP https://files.transkribus.eu/Get?id=SVZAIDARQWLMQTKQVIOOSAKV https://files.transkribus.eu/Get?id=XADEUHKOGGIWCOCZMTGGFHDP https://files.transkribus.eu/Get?id=QRUEXOAFCJBJXSXVTYKDCVKF https://files.transkribus.eu/Get?id=TZWAQKCGYPBWVZNSHVAKWCLZ https://files.transkribus.eu/Get?id=HVTGPLBHUEUXGJMGBNHIJIVS https://files.transkribus.eu/Get?id=KGTWYGJEQVUTLYQJJHDYRSDR https://files.transkribus.eu/Get?id=UCCYYBWUNOZJTXPTLDSGMKED https://files.transkribus.eu/Get?id=FTRLAADUOFFQLRZKSCULMDKF https://files.transkribus.eu/Get?id=BYLJWEGQKLYXLDPIEERCHAHG https://files.transkribus.eu/Get?id=SYZATLLCHFOLZWDODQRXNKNF https://files.transkribus.eu/Get?id=WZJQYUAGMQINLJKWSLBKSGTG https://files.transkribus.eu/Get?id=WOTWYEYWCOLUIZDUZMKQOBPA https://files.transkribus.eu/Get?id=IZHOVYUENLZPLEPYGPDACJXR https://files.transkribus.eu/Get?id=ITMLHBAAKCXDNTLMKMXNWOVM https://files.transkribus.eu/Get?id=GTEWMTDGWLEQMNDTONNPWVAV https://files.transkribus.eu/Get?id=XHVQSIUMEGUOGCMLVIBNPDIZ https://files.transkribus.eu/Get?id=IYDRMOZDVMAEHZLCLYWJFKZU https://files.transkribus.eu/Get?id=OZGAGQKATLEVBZMKNPUSZJZA https://files.transkribus.eu/Get?id=BMYKTWTVVTYYYQQSTXWQHKLC https://files.transkribus.eu/Get?id=IYUGXNCSKYCGDZZATBMXIPRX https://files.transkribus.eu/Get?id=BBTDFLKQWLYLTYYEMEYWRCFX
# Get XML files.
wget --content-disposition https://files.transkribus.eu/Get?id=JHVUSEFVRTWKTDYBEFVBEPFB https://files.transkribus.eu/Get?id=BYFBODHLFRJDQBVOROTLEWET https://files.transkribus.eu/Get?id=RWSSDKJHNAPTDXVLREBGBCLW https://files.transkribus.eu/Get?id=YEPDJBAQSQUMXTBRNXGGFREU https://files.transkribus.eu/Get?id=FZQWWFZCNWCYAPQRVIZTAOSX https://files.transkribus.eu/Get?id=IMXDAERWVVGYBIOMIEFTXFMK https://files.transkribus.eu/Get?id=LVZAFONLFVUFHYVDCAHAERYP https://files.transkribus.eu/Get?id=CIMPTJQYTQESVVGCBQRALMEX https://files.transkribus.eu/Get?id=CVSIQYLKNTXSVXRHBRBHIKKM https://files.transkribus.eu/Get?id=UIOFSVNICSMOSBKPJBZOATHM https://files.transkribus.eu/Get?id=NIEOIIUKDXUNFYJHHXADOUZH https://files.transkribus.eu/Get?id=RECFLPURJRTWVCNLRPARZWCT https://files.transkribus.eu/Get?id=BIIMISGELCYEAVCSFBEOBEXE https://files.transkribus.eu/Get?id=YSCXXSWLKELAWMDBRLAXKZPM https://files.transkribus.eu/Get?id=PPKQXCNQTIPCWWKKYEYKKFWR https://files.transkribus.eu/Get?id=TMGWJSEMDVZZUHIMCJMBQSAS https://files.transkribus.eu/Get?id=RQQGZJRNLALMLBHXWZAQRJJB https://files.transkribus.eu/Get?id=NKRITEGYTBOJEQKHIYFVPIKG https://files.transkribus.eu/Get?id=OHMVVRMMPUDTYDRDFHJZPKHZ https://files.transkribus.eu/Get?id=IQWIGWEOZAOZGEBTROXULIFN https://files.transkribus.eu/Get?id=XKLRDXMZFCGBBDKXXWQUOGTP https://files.transkribus.eu/Get?id=KBBFFENIEDVCCEJKGINYYOIA https://files.transkribus.eu/Get?id=VHHUSIBLNZVDYJDEUGTFNFIB https://files.transkribus.eu/Get?id=NWOLIMGYYSFJGHASEFSAHPSU https://files.transkribus.eu/Get?id=CMXZGEPQCEQPOYXCVMUAAMXL https://files.transkribus.eu/Get?id=UJMUBEOLKJSRNPSVFBSPIIQE https://files.transkribus.eu/Get?id=QNKLLUDVYNWHQFYPDGDKOSWG https://files.transkribus.eu/Get?id=UBELSVTLYVTCJSWPBNAKXCWN https://files.transkribus.eu/Get?id=UAQBIBAIRXGBSPALZPERMJQN https://files.transkribus.eu/Get?id=ABGYGICZRGMJPDKBTQRFMZWY https://files.transkribus.eu/Get?id=IEOAQAMGVZGEDZPSJCWPXJOW https://files.transkribus.eu/Get?id=JLXONNHBYYYPWIACEFKYYIGW https://files.transkribus.eu/Get?id=VJUZWLEDNECLNPBDKLPQIXPP https://files.transkribus.eu/Get?id=IWOOMDSPRWIHAUSGCDHMDRDY https://files.transkribus.eu/Get?id=NNHSAVUBAHYCJNDDFERAQIPK https://files.transkribus.eu/Get?id=ORCJXYGXILWSHLOOHPSECMQL https://files.transkribus.eu/Get?id=DDJGAXYOUQPGAQUBYVKNPSZI https://files.transkribus.eu/Get?id=ONNXZWVCJEGUTVJFDUVLSPIG https://files.transkribus.eu/Get?id=IIAVPDPIIJCOHAZDKJDBAMHL https://files.transkribus.eu/Get?id=VNVJLQRTQLJKWIUJPFDMRNPT https://files.transkribus.eu/Get?id=UYHHWDUMYRMHUMQXPBAWJFFA https://files.transkribus.eu/Get?id=YSSLAKWGNUJKXPMFGMFCHPNW https://files.transkribus.eu/Get?id=AXPZFHVZZQROPNGGQEYZNZXO https://files.transkribus.eu/Get?id=HHQOOTMTFVPDCARHHHGBIAUX https://files.transkribus.eu/Get?id=POTHXBKQPCHCXROHCSKQIWBJ https://files.transkribus.eu/Get?id=DSCRKYNYRSZFFHCQSDFCYUCO https://files.transkribus.eu/Get?id=JZPDYBTUFJAFSUACHHIZBWXH https://files.transkribus.eu/Get?id=YOECZHMWXUKAVVZYRONOQLJX https://files.transkribus.eu/Get?id=URNYUAFDAUHQPAHFMUUBTKHT https://files.transkribus.eu/Get?id=REFTLJUUJNMDPSVNZTSBEJEB https://files.transkribus.eu/Get?id=XMUKGQJUGFOGGGPLQNECYVBK https://files.transkribus.eu/Get?id=TCPFSFMTPGDZEWSMJZBXQEGH https://files.transkribus.eu/Get?id=YHMLVPJHADPBXBWKVJFDNTEL https://files.transkribus.eu/Get?id=JYXIBXPCZESDNNSHCBETTLLB https://files.transkribus.eu/Get?id=RQVWZBXVHZHRBPDJLSYDINFB https://files.transkribus.eu/Get?id=KIUANWLYIDEJYTDSVUFAWUCV https://files.transkribus.eu/Get?id=POZKJCZOPOKGILWSBPFUYDNK https://files.transkribus.eu/Get?id=PLIDQKLLDDSTURVBSYUKEHOJ https://files.transkribus.eu/Get?id=IRPJIVREFPZBZYXFELHDZWTH https://files.transkribus.eu/Get?id=GCAWPLAWQVTKWUDKTXBCTRJY https://files.transkribus.eu/Get?id=LTGBEPJCSYDUNHIIFKLVUJQX https://files.transkribus.eu/Get?id=AYRNLZJFWCBVZOFUNVQBTDYE https://files.transkribus.eu/Get?id=FHLBDWEYSNDJJZOAOBJTSVBU https://files.transkribus.eu/Get?id=DLDNAFADKWZLSBRCBLVCXCSL https://files.transkribus.eu/Get?id=YQTEGXGRQLFNNOYEUWBFRYWL https://files.transkribus.eu/Get?id=XHVQSESDDDCAALHGRMUBTXHV https://files.transkribus.eu/Get?id=RQLANQXXACUDSKBSYKIQNODK https://files.transkribus.eu/Get?id=FYQCLMNREDKIKKOUOOLGWEAX https://files.transkribus.eu/Get?id=UNFTXZTWQDELXHLHXDZSDWYX https://files.transkribus.eu/Get?id=ORBMIMGYCXLYQMYIAEIOXPJV https://files.transkribus.eu/Get?id=VEITMPANUCBOUFALNYCRWHTI https://files.transkribus.eu/Get?id=ECNNPBOIQRTBDNLSPXDMPGLP https://files.transkribus.eu/Get?id=QKCSSFYBNNVCOSFIADEDCXDA https://files.transkribus.eu/Get?id=LBPAHTIOKUBENKAYUXZECLSN https://files.transkribus.eu/Get?id=UXPPULKUSALSNGREBVUEWAHK https://files.transkribus.eu/Get?id=EIPMIKZZMACPQAYNQHTJCNKX https://files.transkribus.eu/Get?id=NFVGPKGIVBNNHLUSALCJKVQD https://files.transkribus.eu/Get?id=DZHPULUWLVNRBEVGOOWJPHAY https://files.transkribus.eu/Get?id=DTPIETEDOPLJBMYMVJRASTVP https://files.transkribus.eu/Get?id=TLGKJAWVGBPWWSUPTXHZYTWA https://files.transkribus.eu/Get?id=VQLTSTPZFUURQZWASJKPPLRR https://files.transkribus.eu/Get?id=TVUSOUXQBJCRBWDGZWKBUJSL https://files.transkribus.eu/Get?id=PXWWTCWWTEWUONCKNUBJSJDQ https://files.transkribus.eu/Get?id=JJBWWXFHRTPDNVEJHVMEEPRO https://files.transkribus.eu/Get?id=BVNFMMNMGEOFCNOXKNKAODQH https://files.transkribus.eu/Get?id=VCZYYLYPGLGDUEEOLTSKGRCP https://files.transkribus.eu/Get?id=YRBFGZUNUWESRFIZAWFZDLXA https://files.transkribus.eu/Get?id=RTMXEFVCUSQYBUFGXJROOSQW https://files.transkribus.eu/Get?id=WHUQMBKMTMBXEAJTENUGAWTY https://files.transkribus.eu/Get?id=EOFPMFLTWMSWKVWNUOOBFPWG https://files.transkribus.eu/Get?id=XQFIWGCRJEASTRDJAEJUCKKD https://files.transkribus.eu/Get?id=KZRXAIFVAXZSUELIMALBPERE https://files.transkribus.eu/Get?id=AOZASWDDOAXNSZAZUUAKZUVW https://files.transkribus.eu/Get?id=NQHOQQTPPIGHMJOXLHZCMKIK https://files.transkribus.eu/Get?id=ZWGLSCBYSFNEBDLBAZUXLAEC https://files.transkribus.eu/Get?id=ZNKWVPEGXHBULCAOEKGGGJKR https://files.transkribus.eu/Get?id=TKANIGGKMDYJJBZJVOMVKBHL https://files.transkribus.eu/Get?id=WVQFNOXILOFEARUYWIHZDCVP https://files.transkribus.eu/Get?id=CGUNPASSZQOASOYNDPAPWUAI https://files.transkribus.eu/Get?id=WFYPPCWSRVAMVXIJROZTESKN https://files.transkribus.eu/Get?id=PLGVGOXDAMGBNCISZQHKBVVJ https://files.transkribus.eu/Get?id=BJXKMZOQAUZCFJGHDKIDIKWY https://files.transkribus.eu/Get?id=CMDZVVKNEJZKHZKZRTRQOLBC https://files.transkribus.eu/Get?id=IUCKTFOEMGCBWQMSNKBJBMLE https://files.transkribus.eu/Get?id=EIZMZATKMGACEAALZFSMTHQJ https://files.transkribus.eu/Get?id=YJLDTJWFIEZYUDLALRXVNIBK https://files.transkribus.eu/Get?id=QSJICLTFSXESUQEFXPMEVACO https://files.transkribus.eu/Get?id=JVEAELRWZRPXATUPJNMPSAIO https://files.transkribus.eu/Get?id=XLISFOZJUAXSOZXKWGTTHKDD https://files.transkribus.eu/Get?id=VVCOOOIHGMRDLSMDHMPGYLFE https://files.transkribus.eu/Get?id=GZEEXENYIDTXCTPBSAORHHSA https://files.transkribus.eu/Get?id=IDGJOMRHAFKFUEHXKQGCUFNG https://files.transkribus.eu/Get?id=UKXMGZJJCWNYHCVKXSOCWTBI https://files.transkribus.eu/Get?id=BARPNPDXBCVDSYLRXQITLSLU https://files.transkribus.eu/Get?id=HTWTCHRHVYIYHVMITIYNPGOZ https://files.transkribus.eu/Get?id=NZSTGDRQKMIECMYFKIXRGWQF https://files.transkribus.eu/Get?id=ICTGHWNQUZZZGZOMQBKWFDIP https://files.transkribus.eu/Get?id=QZYDJWWQDHCWRRWCIKBFHLVB https://files.transkribus.eu/Get?id=XDUTNFFNKXYTPZGFHFLEHTSY https://files.transkribus.eu/Get?id=ISXNKHWMHCVPRJLHBZPOWXNR https://files.transkribus.eu/Get?id=QXXAZLBIQPTNGSQBIAYVNGWB https://files.transkribus.eu/Get?id=NYSQTMYOGWTKDEPTPKNCJUWC https://files.transkribus.eu/Get?id=EONVKVKOIAHEROQVYEQZIXKR https://files.transkribus.eu/Get?id=XVNYPROHFPLWSUQKMCBYVOTI https://files.transkribus.eu/Get?id=HUDJOFZWWGELFRBJVVZNRJAS https://files.transkribus.eu/Get?id=FFDRJLMVQTQGUITZNOWSRFHT https://files.transkribus.eu/Get?id=OIAVJGZJSNRVPIQOQGKELTTS https://files.transkribus.eu/Get?id=NKYVSVOAKQUOCKKVRYHBGZYH https://files.transkribus.eu/Get?id=KJNSHSVFLRTXPBRLABNVJNEZ https://files.transkribus.eu/Get?id=MBSKPVBMWWDUAAXUEJDYNLPZ https://files.transkribus.eu/Get?id=MHLGDEADRYDIURDNKUFEGPOB https://files.transkribus.eu/Get?id=CTDTEHIJXZCTRQOMLHCQAJIE https://files.transkribus.eu/Get?id=SNZPGLFMDJIHJTIJWPUWMNLO https://files.transkribus.eu/Get?id=NICXUBXEUYOWGTCAOHUXLQYP https://files.transkribus.eu/Get?id=YKDBOQISRUEPRNOGCQRJSKZO https://files.transkribus.eu/Get?id=DCKSZDCZAELVYSCOGSYMZCRV https://files.transkribus.eu/Get?id=JAZYPAUWVLULYUMXQXSXPDUP https://files.transkribus.eu/Get?id=KJFYLNDRPHMNWWNBRASILZSO https://files.transkribus.eu/Get?id=GXZXDWEKOUMRRWVGCKBVOYDX https://files.transkribus.eu/Get?id=XXASBUPYNMROTAPFYCPEKTFO https://files.transkribus.eu/Get?id=UTWJIKOCOWILGTHAVQELNXZP https://files.transkribus.eu/Get?id=UQWTMHFCAGGNCRPDHMISHEZQ https://files.transkribus.eu/Get?id=QOLIGHKUCUPILRHJZTQBMAQW https://files.transkribus.eu/Get?id=QKGCVEHTCAHRULIYFKOKKCFY https://files.transkribus.eu/Get?id=KHQDPSYPSBRGAJUFRJMOYYHV https://files.transkribus.eu/Get?id=VKWKDCOHWOCNAROYRJKHLFYC https://files.transkribus.eu/Get?id=XNLDKCGFMCHVEZTAJQLOHVYS https://files.transkribus.eu/Get?id=LSUTSNAWNILOCLCJLJVHHPPP https://files.transkribus.eu/Get?id=VBICXRUQAIGVEXAVMWTWIUDK https://files.transkribus.eu/Get?id=KJFZNMXHNIVVXNRWRLNPFOOT https://files.transkribus.eu/Get?id=JBOQSXZLWOLKVMLXOVQDOKJU https://files.transkribus.eu/Get?id=IMDEJHWZKACFXIGKEFXPQLII https://files.transkribus.eu/Get?id=QZKWGDEXJPZKTDUCDFWDEOET https://files.transkribus.eu/Get?id=EQSKRRFMYFDNFAJKYSBEAGQZ https://files.transkribus.eu/Get?id=ECTQOJPXUKYFSXVWRRSGVQRO https://files.transkribus.eu/Get?id=XRDHUZNHGKFBUMJFWMZSNYGM https://files.transkribus.eu/Get?id=VXBCUSYDWISJNFPVLOPGMHQL https://files.transkribus.eu/Get?id=HASZIIPNMFNRBGRFNAHLLUBR https://files.transkribus.eu/Get?id=MUKOMKPEEGXLSMZSNNZVZQVF https://files.transkribus.eu/Get?id=TEYUPVVQHWBKCNMEBDOYBTNY https://files.transkribus.eu/Get?id=YYEOVUJKGVARJPNSIFGLTJDA https://files.transkribus.eu/Get?id=OZBOZQQGNFNCAIZRFBXBZTQU https://files.transkribus.eu/Get?id=SPDIYTHJUBLTXUFRDZHXVVNF https://files.transkribus.eu/Get?id=GBDRCZDQRFRWUHIBQMQQFWOB https://files.transkribus.eu/Get?id=PUXPSBYVMNNKUVFDVBXBUFGU https://files.transkribus.eu/Get?id=VWQGXYKCTNVODXWYFXCXIESE https://files.transkribus.eu/Get?id=QAGEQHUIBPCKCVZKCAYSKJAS https://files.transkribus.eu/Get?id=DXHLEYZKFOUPCGDJKILGLEEN https://files.transkribus.eu/Get?id=XJIGEZRWLZCHXNFMCJVGRJYO https://files.transkribus.eu/Get?id=CDFNKPNPFNLUDLMHNSVWIOJY https://files.transkribus.eu/Get?id=GMJIJQMKUDSKYOZBYNUSAXNB https://files.transkribus.eu/Get?id=ZIKEBIHOCZEPWRSXGLQIFANV https://files.transkribus.eu/Get?id=UMXWNECUPMBVUSYXQSQTLJXA https://files.transkribus.eu/Get?id=DOYOKEEZUPSEPOLQFFOSDXLI https://files.transkribus.eu/Get?id=BXVPCUAOJBDIXZTHJNVJQUVG https://files.transkribus.eu/Get?id=ZKGAJZFHGCNQPJACKBVUONZZ https://files.transkribus.eu/Get?id=KAAROZFEMHTPFBBZRROKOWBF https://files.transkribus.eu/Get?id=KGDGOZQBVHZIPFABKPUYMOPG https://files.transkribus.eu/Get?id=QEJFWXTMXGFJTFWCORHVCGCR https://files.transkribus.eu/Get?id=LPDOYQEQTNASZIWTBRVUZILS https://files.transkribus.eu/Get?id=YLQKKITJQKAOCOHWYOTYZUMC https://files.transkribus.eu/Get?id=LOTMIHEOLSGYTTJQPJCUDXIF https://files.transkribus.eu/Get?id=NLDJIZLJUKUYPWQZHQPRBACG https://files.transkribus.eu/Get?id=PNLSPNFUBYEFSRAMKEGHZRMX https://files.transkribus.eu/Get?id=TFLTRJJTYRGVLRUZKWXQJTSL https://files.transkribus.eu/Get?id=IGKKGDXZEUPHQEHRNVGGGSHF https://files.transkribus.eu/Get?id=KDNTAWHVDCRMWUJYHCXXNAEJ https://files.transkribus.eu/Get?id=CJNSNRLHKBUNFWVVDPFBOYGC https://files.transkribus.eu/Get?id=OZJOAVTDJANNGFWRCSPSSMNT https://files.transkribus.eu/Get?id=ZAKYZRJPMEUGAFMEETNWJHYD https://files.transkribus.eu/Get?id=AAKZVULGHVARRSMQCUZFRQOT https://files.transkribus.eu/Get?id=ANJKLTOEBRJAOFAZIQAOPDVL https://files.transkribus.eu/Get?id=QQENWYLYGEORJOOWXQGDDOYQ https://files.transkribus.eu/Get?id=RWKPNTKIKQTIMRNWWYGVBHXM https://files.transkribus.eu/Get?id=IAOLLMALNIHPDWGDSPCAKUFB https://files.transkribus.eu/Get?id=TOZWZVEROLTDQSSGQVVIDPFW https://files.transkribus.eu/Get?id=NZAQUQJMEOMMALKIAJTKYCEB https://files.transkribus.eu/Get?id=IYTYIEJULURSVZXZJOKNLGTP https://files.transkribus.eu/Get?id=PDDUAQELWDADQVLKYKTMUIEO
cd ..
mkdir Validation
cd Validation
# Get image files.
wget --content-disposition https://files.transkribus.eu/Get?id=VKHSBYKDRSPCXNYVVDUUJHIJ https://files.transkribus.eu/Get?id=ZFDKMFYHZSSWJRWJDBSKXLME https://files.transkribus.eu/Get?id=KUNKUUMWRVICWVMISJETKKSP https://files.transkribus.eu/Get?id=VCYDPLIKHZNSMBXZNEARDMQG https://files.transkribus.eu/Get?id=DSZCVDYESIQHIKUWNBMEGUMI https://files.transkribus.eu/Get?id=UMTPCFPNQPEMZGGSGOZWRHZZ https://files.transkribus.eu/Get?id=XGLLYKKRWGKDJTSTQZTQSBDB https://files.transkribus.eu/Get?id=GMDPHGGENKKZXGUNAYDUYWLO https://files.transkribus.eu/Get?id=YWGNYDFWVHJLFFSYFHFOPPWV https://files.transkribus.eu/Get?id=IVUIBDCTBUYBGAEWVULXXFST https://files.transkribus.eu/Get?id=SQCNQZEQTSUHTQWYPFYQALUH https://files.transkribus.eu/Get?id=SCOYGUKQFUEPENAKBDSOXYGF https://files.transkribus.eu/Get?id=UHGKWLEZJGEGEUDTJWHQHWGT https://files.transkribus.eu/Get?id=RLXCXHKMEZPDOPKXEQWALGPY https://files.transkribus.eu/Get?id=REUHQRVBWHKVAGYKNOBKCSNP https://files.transkribus.eu/Get?id=FARONPPATEJYGRXVRHXRBPPI https://files.transkribus.eu/Get?id=BYXLCAPXMRURUGUHGOCDUAWJ https://files.transkribus.eu/Get?id=RHZLBFIAGXPNVSEKDMXVNAZT https://files.transkribus.eu/Get?id=HMXEFFBXKKVTASJBJNLMYPNL https://files.transkribus.eu/Get?id=JRNXMKFSVZEOMVCMPGWFRPYJ https://files.transkribus.eu/Get?id=NTGOQDNHUKAKGEUWVHRHDUOR https://files.transkribus.eu/Get?id=JJLLOATKBIRFPVVUMQETOTTN https://files.transkribus.eu/Get?id=CAOYGGOXCZVJYMBKTZGNCFCT https://files.transkribus.eu/Get?id=WSAKKKUEQCSGPKCLCVHHCKVD
# Get XML files.
wget --content-disposition https://files.transkribus.eu/Get?id=CHDVOIIYLHWZEPXMAUJWWNKU https://files.transkribus.eu/Get?id=ZRLEVECHRBPRETNOKXOFWGPU https://files.transkribus.eu/Get?id=WCIRBUXVTCXPGZTDSGJLGVWR https://files.transkribus.eu/Get?id=LMXKCJUHLQTOTWZKQOYBXRRH https://files.transkribus.eu/Get?id=RVNPPKIXIUTUFYAYGDUMWPRB https://files.transkribus.eu/Get?id=BOCCILWYTULGOIMRGCUHLLBF https://files.transkribus.eu/Get?id=PHXCDTHIHWHPCPVZANZLMEZW https://files.transkribus.eu/Get?id=ERCWXYVDDXSQTNDNRADLLNSL https://files.transkribus.eu/Get?id=VWCEXDWQJWMZPPBWPGVUFLSU https://files.transkribus.eu/Get?id=REJFIMVSKOBVFYCGVPRYXSIL https://files.transkribus.eu/Get?id=QPDBOUSCBUNSAVIKCHKWKLOP https://files.transkribus.eu/Get?id=XVCPFSVVGHVLQYTRGUPECDBO https://files.transkribus.eu/Get?id=KAHBRRJHNECLTFIDYKEVHIKN https://files.transkribus.eu/Get?id=WPWCKGCRUCANVXXOAXMMMJHM https://files.transkribus.eu/Get?id=XQUUXHYOWQGAUAQIYITHIMBX https://files.transkribus.eu/Get?id=XJYJHKBQWTLIIBGYXZYMOMTE https://files.transkribus.eu/Get?id=RWJTCQCXVXQOTBIMHLYRZXYN https://files.transkribus.eu/Get?id=JBLGRLYTPUUOYLKCPNNSMGCY https://files.transkribus.eu/Get?id=AYSTOXAKCOUKGCLBAGEDTBQN https://files.transkribus.eu/Get?id=KRNQGZWNGROPPIIJXPATHOHH https://files.transkribus.eu/Get?id=QDARVKDOIPYISUFZBCOTHCXV https://files.transkribus.eu/Get?id=EMFJCMSQWEELQTLIQDVDHBHJ https://files.transkribus.eu/Get?id=FDIATJURTJQLLGMJTHTQIQTC https://files.transkribus.eu/Get?id=CUMMKSQMFFWKWEEBFRYKWNRL
cd ..
Install kraken
python3.9 -m venv venv3.9
source venv3.9/bin/activate
pip install -U pip setuptools wheel
pip install kraken
pip install -U torch torchvision --extra-index-url https://download.pytorch.org/whl/cu113
Run training
The training is running on a Debian GNU Linux server with Nvidia RTX A5000 GPU.
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen -d cuda:0 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/24/22 17:38:52] WARNING alphabet mismatch: chars in training set only: {'โก', '[', 'โ ', '๊', '=', 'รป', '๊ธ', 'ยบ', 'X', 'โ', 'ร', 'โ', 'โ', 'ยฝ', ']', train.py:304
'๊ฏ'} (not included in accuracy test during training)
WARNING alphabet mismatch: chars in validation set only: {'ร', 'รน', 'อฆ'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/24/22 17:38:54] WARNING Non-encodable sequence รนltis... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence รrโ... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence อฆ 166... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
โโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ โ Name โ Type โ Params โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 0 โ net โ MultiParamSequential โ 4.0 M โ
โ 1 โ net.C_0 โ ActConv2D โ 1.3 K โ
โ 2 โ net.Do_1 โ Dropout โ 0 โ
โ 3 โ net.Mp_2 โ MaxPool โ 0 โ
โ 4 โ net.C_3 โ ActConv2D โ 40.0 K โ
โ 5 โ net.Do_4 โ Dropout โ 0 โ
โ 6 โ net.Mp_5 โ MaxPool โ 0 โ
โ 7 โ net.C_6 โ ActConv2D โ 55.4 K โ
โ 8 โ net.Do_7 โ Dropout โ 0 โ
โ 9 โ net.Mp_8 โ MaxPool โ 0 โ
โ 10 โ net.C_9 โ ActConv2D โ 110 K โ
โ 11 โ net.Do_10 โ Dropout โ 0 โ
โ 12 โ net.S_11 โ Reshape โ 0 โ
โ 13 โ net.L_12 โ TransposedSummarizingRNN โ 1.9 M โ
โ 14 โ net.Do_13 โ Dropout โ 0 โ
โ 15 โ net.L_14 โ TransposedSummarizingRNN โ 963 K โ
โ 16 โ net.Do_15 โ Dropout โ 0 โ
โ 17 โ net.L_16 โ TransposedSummarizingRNN โ 963 K โ
โ 18 โ net.Do_17 โ Dropout โ 0 โ
โ 19 โ net.O_18 โ LinSoftmax โ 48.5 K โ
โโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:58 val_accuracy: 0.00000 early_stopping: 0/20 0.00000
stage 1/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:57 val_accuracy: 0.27963 early_stopping: 0/20 0.27963
stage 2/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:44 val_accuracy: 0.63872 early_stopping: 0/20 0.63872
stage 3/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:02 val_accuracy: 0.77172 early_stopping: 0/20 0.77172
stage 4/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:59 val_accuracy: 0.82858 early_stopping: 0/20 0.82858
stage 5/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:05 val_accuracy: 0.86218 early_stopping: 0/20 0.86218
stage 6/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:50 val_accuracy: 0.88192 early_stopping: 0/20 0.88192
stage 7/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:01 val_accuracy: 0.89528 early_stopping: 0/20 0.89528
stage 8/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:55 val_accuracy: 0.90674 early_stopping: 0/20 0.90674
stage 9/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:02 val_accuracy: 0.90923 early_stopping: 0/20 0.90923
stage 10/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:03 val_accuracy: 0.91599 early_stopping: 0/20 0.91599
stage 11/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:48 val_accuracy: 0.91963 early_stopping: 0/20 0.91963
stage 12/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:42 val_accuracy: 0.92259 early_stopping: 0/20 0.92259
stage 13/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:35 val_accuracy: 0.92418 early_stopping: 0/20 0.92418
stage 14/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:59 val_accuracy: 0.92860 early_stopping: 0/20 0.92860
stage 15/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:46 val_accuracy: 0.92850 early_stopping: 1/20 0.92860
stage 16/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:48 val_accuracy: 0.92978 early_stopping: 0/20 0.92978
stage 17/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:49 val_accuracy: 0.93181 early_stopping: 0/20 0.93181
stage 18/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:02 val_accuracy: 0.93016 early_stopping: 1/20 0.93181
stage 19/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:55 val_accuracy: 0.93448 early_stopping: 0/20 0.93448
stage 20/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:33 val_accuracy: 0.93439 early_stopping: 1/20 0.93448
stage 21/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:27 val_accuracy: 0.93539 early_stopping: 0/20 0.93539
stage 22/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:33 val_accuracy: 0.93567 early_stopping: 0/20 0.93567
stage 23/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:32 val_accuracy: 0.93588 early_stopping: 0/20 0.93588
stage 24/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:39 val_accuracy: 0.93722 early_stopping: 0/20 0.93722
stage 25/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:41 val_accuracy: 0.93679 early_stopping: 1/20 0.93722
stage 26/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:31 val_accuracy: 0.93707 early_stopping: 2/20 0.93722
stage 27/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:34 val_accuracy: 0.93847 early_stopping: 0/20 0.93847
stage 28/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:32 val_accuracy: 0.93747 early_stopping: 1/20 0.93847
stage 29/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:42 val_accuracy: 0.93884 early_stopping: 0/20 0.93884
stage 30/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:23 val_accuracy: 0.93884 early_stopping: 1/20 0.93884
stage 31/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:21 val_accuracy: 0.93682 early_stopping: 2/20 0.93884
stage 32/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:44 val_accuracy: 0.93869 early_stopping: 3/20 0.93884
stage 33/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:49 val_accuracy: 0.93657 early_stopping: 4/20 0.93884
stage 34/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:37 val_accuracy: 0.93919 early_stopping: 0/20 0.93919
stage 35/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:39 val_accuracy: 0.93782 early_stopping: 1/20 0.93919
stage 36/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:04:27 val_accuracy: 0.93875 early_stopping: 2/20 0.93919
stage 37/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:49 val_accuracy: 0.94028 early_stopping: 0/20 0.94028
stage 38/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:05:50 val_accuracy: 0.94112 early_stopping: 0/20 0.94112
stage 39/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:06:05 val_accuracy: 0.93978 early_stopping: 1/20 0.94112
stage 40/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:08:45 val_accuracy: 0.93922 early_stopping: 2/20 0.94112
stage 41/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:09:56 val_accuracy: 0.94046 early_stopping: 3/20 0.94112
stage 42/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:07:38 val_accuracy: 0.94186 early_stopping: 0/20 0.94186
stage 43/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:09:34 val_accuracy: 0.93940 early_stopping: 1/20 0.94186
stage 44/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:09:55 val_accuracy: 0.93909 early_stopping: 2/20 0.94186
stage 45/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 1:10:04 val_accuracy: 0.94040 early_stopping: 3/20 0.94186
stage 46/โ โโโโโโโโโโโบโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 1930/7366 0:51:25 0:18:01 val_accuracy: 0.94040 early_stopping: 3/20 0.94186
Run training with augmentation
Using more than one worker process makes the training much faster, and augmentation should result in a better model.
pip install albumentations
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+ -d cuda:0 --augment --workers 4 -r 0.0001 -B 1 --min-epochs 20 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/26/22 13:43:36] WARNING alphabet mismatch: chars in training set only: {'โ', '๊ฏ', 'รป', ']', '๊', '๊ธ', 'ร', '[', 'โ ', '=', 'โก', 'ยบ', 'ยฝ', 'X', 'โ', 'โ'} (not included in accuracy test during train.py:304
training)
WARNING alphabet mismatch: chars in validation set only: {'อฆ', 'ร', 'รน'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/26/22 13:43:38] WARNING Non-encodable sequence รนltis... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence รrโ... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence อฆ 166... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
โโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ โ Name โ Type โ Params โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 0 โ net โ MultiParamSequential โ 4.0 M โ
โ 1 โ net.C_0 โ ActConv2D โ 1.3 K โ
โ 2 โ net.Do_1 โ Dropout โ 0 โ
โ 3 โ net.Mp_2 โ MaxPool โ 0 โ
โ 4 โ net.C_3 โ ActConv2D โ 40.0 K โ
โ 5 โ net.Do_4 โ Dropout โ 0 โ
โ 6 โ net.Mp_5 โ MaxPool โ 0 โ
โ 7 โ net.C_6 โ ActConv2D โ 55.4 K โ
โ 8 โ net.Do_7 โ Dropout โ 0 โ
โ 9 โ net.Mp_8 โ MaxPool โ 0 โ
โ 10 โ net.C_9 โ ActConv2D โ 110 K โ
โ 11 โ net.Do_10 โ Dropout โ 0 โ
โ 12 โ net.S_11 โ Reshape โ 0 โ
โ 13 โ net.L_12 โ TransposedSummarizingRNN โ 1.9 M โ
โ 14 โ net.Do_13 โ Dropout โ 0 โ
โ 15 โ net.L_14 โ TransposedSummarizingRNN โ 963 K โ
โ 16 โ net.Do_15 โ Dropout โ 0 โ
โ 17 โ net.L_16 โ TransposedSummarizingRNN โ 963 K โ
โ 18 โ net.Do_17 โ Dropout โ 0 โ
โ 19 โ net.O_18 โ LinSoftmax โ 48.5 K โ
โโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/โ โโโโธโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 673/7366 0:16:39 0:01:43 early_stopping: 0/5 -inf
Increasing the number of workers further reduces the time per epoch even more.
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+ -d cuda:0
--augment --workers 16 -r 0.0001 -B 1 --min-epochs 20 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/26/22 13:49:52] WARNING alphabet mismatch: chars in training set only: {'๊ฏ', 'ยฝ', '๊', 'โ', '=', '[', 'รป', 'โ', 'ร', 'โ ', 'โ', '๊ธ', 'โก', 'X', 'ยบ', ']'} (not included in accuracy test during train.py:304
training)
WARNING alphabet mismatch: chars in validation set only: {'ร', 'รน', 'อฆ'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/26/22 13:49:55] WARNING Non-encodable sequence รนltis... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence รrโ... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence อฆ 166... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
โโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ โ Name โ Type โ Params โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 0 โ net โ MultiParamSequential โ 4.0 M โ
โ 1 โ net.C_0 โ ActConv2D โ 1.3 K โ
โ 2 โ net.Do_1 โ Dropout โ 0 โ
โ 3 โ net.Mp_2 โ MaxPool โ 0 โ
โ 4 โ net.C_3 โ ActConv2D โ 40.0 K โ
โ 5 โ net.Do_4 โ Dropout โ 0 โ
โ 6 โ net.Mp_5 โ MaxPool โ 0 โ
โ 7 โ net.C_6 โ ActConv2D โ 55.4 K โ
โ 8 โ net.Do_7 โ Dropout โ 0 โ
โ 9 โ net.Mp_8 โ MaxPool โ 0 โ
โ 10 โ net.C_9 โ ActConv2D โ 110 K โ
โ 11 โ net.Do_10 โ Dropout โ 0 โ
โ 12 โ net.S_11 โ Reshape โ 0 โ
โ 13 โ net.L_12 โ TransposedSummarizingRNN โ 1.9 M โ
โ 14 โ net.Do_13 โ Dropout โ 0 โ
โ 15 โ net.L_14 โ TransposedSummarizingRNN โ 963 K โ
โ 16 โ net.Do_15 โ Dropout โ 0 โ
โ 17 โ net.L_16 โ TransposedSummarizingRNN โ 963 K โ
โ 18 โ net.Do_17 โ Dropout โ 0 โ
โ 19 โ net.O_18 โ LinSoftmax โ 48.5 K โ
โโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.00000 early_stopping: 0/5 0.00000
stage 1/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.20782 early_stopping: 0/5 0.20782
stage 2/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.57978 early_stopping: 0/5 0.57978
stage 3/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.71006 early_stopping: 0/5 0.71006
stage 4/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:20 val_accuracy: 0.78446 early_stopping: 0/5 0.78446
stage 5/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.82139 early_stopping: 0/5 0.82139
stage 6/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:57 val_accuracy: 0.83555 early_stopping: 0/5 0.83555
stage 7/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.86224 early_stopping: 0/5 0.86224
stage 8/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.86741 early_stopping: 0/5 0.86741
stage 9/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.88089 early_stopping: 0/5 0.88089
stage 10/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.89192 early_stopping: 0/5 0.89192
stage 11/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.87604 early_stopping: 1/5 0.89192
stage 12/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.89733 early_stopping: 0/5 0.89733
stage 13/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.90216 early_stopping: 0/5 0.90216
stage 14/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.90222 early_stopping: 0/5 0.90222
stage 15/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.90182 early_stopping: 1/5 0.90222
stage 16/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:48 val_accuracy: 0.90608 early_stopping: 0/5 0.90608
stage 17/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:44 val_accuracy: 0.91259 early_stopping: 0/5 0.91259
stage 18/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.90496 early_stopping: 1/5 0.91259
stage 19/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91141 early_stopping: 2/5 0.91259
stage 20/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.90126 early_stopping: 3/5 0.91259
stage 21/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91119 early_stopping: 4/5 0.91259
stage 22/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91100 early_stopping: 5/5 0.91259
Moving best model Juristische_Konsilien_Tuebingen+_17.mlmodel (0.9125926494598389) to Juristische_Konsilien_Tuebingen+_best.mlmodel
real 151m8.293s
user 2131m47.894s
sys 1193m39.862s
Resume the training:
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -i Juristische_Konsilien_Tuebingen+_17.mlmodel -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+ -d cuda:0 --augment --workers 16 -r 0.0001 -B 1 --min-epochs 200 --lag 10 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0
.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/26/22 16:56:31] WARNING alphabet mismatch: chars in training set only: {'ร', 'โก', 'รป', 'โ', 'โ', '๊ฏ', 'ยบ', '๊ธ', 'โ ', '=', 'X', '[', 'ยฝ', ']', '๊', 'โ'} (not included in accuracy test during train.py:304
training)
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
โโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ โ Name โ Type โ Params โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 0 โ net โ MultiParamSequential โ 4.0 M โ
โ 1 โ net.C_0 โ ActConv2D โ 1.3 K โ
โ 2 โ net.Do_1 โ Dropout โ 0 โ
โ 3 โ net.Mp_2 โ MaxPool โ 0 โ
โ 4 โ net.C_3 โ ActConv2D โ 40.0 K โ
โ 5 โ net.Do_4 โ Dropout โ 0 โ
โ 6 โ net.Mp_5 โ MaxPool โ 0 โ
โ 7 โ net.C_6 โ ActConv2D โ 55.4 K โ
โ 8 โ net.Do_7 โ Dropout โ 0 โ
โ 9 โ net.Mp_8 โ MaxPool โ 0 โ
โ 10 โ net.C_9 โ ActConv2D โ 110 K โ
โ 11 โ net.Do_10 โ Dropout โ 0 โ
โ 12 โ net.S_11 โ Reshape โ 0 โ
โ 13 โ net.L_12 โ TransposedSummarizingRNN โ 1.9 M โ
โ 14 โ net.Do_13 โ Dropout โ 0 โ
โ 15 โ net.L_14 โ TransposedSummarizingRNN โ 963 K โ
โ 16 โ net.Do_15 โ Dropout โ 0 โ
โ 17 โ net.L_16 โ TransposedSummarizingRNN โ 963 K โ
โ 18 โ net.Do_17 โ Dropout โ 0 โ
โ 19 โ net.O_18 โ LinSoftmax โ 48.5 K โ
โโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:05 val_accuracy: 0.91661 early_stopping: 0/10 0.91661
stage 1/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.89668 early_stopping: 1/10 0.91661
stage 2/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.91540 early_stopping: 2/10 0.91661
stage 3/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92044 early_stopping: 0/10 0.92044
stage 4/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:19 val_accuracy: 0.90824 early_stopping: 1/10 0.92044
stage 5/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91755 early_stopping: 2/10 0.92044
stage 6/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.90014 early_stopping: 3/10 0.92044
stage 7/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.91244 early_stopping: 4/10 0.92044
stage 8/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.91060 early_stopping: 5/10 0.92044
stage 9/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:02 val_accuracy: 0.91213 early_stopping: 6/10 0.92044
stage 10/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.90397 early_stopping: 7/10 0.92044
stage 11/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92128 early_stopping: 0/10 0.92128
stage 12/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.90992 early_stopping: 1/10 0.92128
stage 13/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92181 early_stopping: 0/10 0.92181
stage 14/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.91278 early_stopping: 1/10 0.92181
stage 15/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92343 early_stopping: 0/10 0.92343
stage 16/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92577 early_stopping: 0/10 0.92577
stage 17/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92166 early_stopping: 1/10 0.92577
stage 18/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.92197 early_stopping: 2/10 0.92577
stage 19/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92240 early_stopping: 3/10 0.92577
stage 20/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:57 val_accuracy: 0.91297 early_stopping: 4/10 0.92577
stage 21/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92078 early_stopping: 5/10 0.92577
stage 22/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92418 early_stopping: 6/10 0.92577
stage 23/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92611 early_stopping: 0/10 0.92611
stage 24/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.91758 early_stopping: 1/10 0.92611
stage 25/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92412 early_stopping: 2/10 0.92611
stage 26/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91991 early_stopping: 3/10 0.92611
stage 27/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92287 early_stopping: 4/10 0.92611
stage 28/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92069 early_stopping: 5/10 0.92611
stage 29/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92767 early_stopping: 0/10 0.92767
stage 30/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:53 val_accuracy: 0.92234 early_stopping: 1/10 0.92767
stage 31/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:41 val_accuracy: 0.92848 early_stopping: 0/10 0.92848
stage 32/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91652 early_stopping: 1/10 0.92848
stage 33/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.92704 early_stopping: 2/10 0.92848
stage 34/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93280 early_stopping: 0/10 0.93280
stage 35/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91870 early_stopping: 1/10 0.93280
stage 36/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92726 early_stopping: 2/10 0.93280
stage 37/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92041 early_stopping: 3/10 0.93280
stage 38/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92673 early_stopping: 4/10 0.93280
stage 39/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92894 early_stopping: 5/10 0.93280
stage 40/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.92633 early_stopping: 6/10 0.93280
stage 41/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.92695 early_stopping: 7/10 0.93280
stage 42/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.91484 early_stopping: 8/10 0.93280
stage 43/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92997 early_stopping: 9/10 0.93280
stage 44/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93296 early_stopping: 0/10 0.93296
stage 45/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91892 early_stopping: 1/10 0.93296
stage 46/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91798 early_stopping: 2/10 0.93296
stage 47/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92888 early_stopping: 3/10 0.93296
stage 48/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93000 early_stopping: 4/10 0.93296
stage 49/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92135 early_stopping: 5/10 0.93296
stage 50/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92256 early_stopping: 6/10 0.93296
stage 51/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92975 early_stopping: 7/10 0.93296
stage 52/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.92468 early_stopping: 8/10 0.93296
stage 53/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.92589 early_stopping: 9/10 0.93296
stage 54/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92589 early_stopping: 9/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 54/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92265 early_stopping: 10/10 0.93296
stage 55/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92265 early_stopping: 10/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 55/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92901 early_stopping: 11/10 0.93296
stage 56/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92901 early_stopping: 11/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 56/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92493 early_stopping: 12/10 0.93296
stage 57/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92493 early_stopping: 12/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 57/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92159 early_stopping: 13/10 0.93296
stage 58/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92159 early_stopping: 13/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 58/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92097 early_stopping: 14/10 0.93296
stage 59/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92097 early_stopping: 14/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 59/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92620 early_stopping: 15/10 0.93296
stage 60/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93495 early_stopping: 0/10 0.93495
stage 61/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92433 early_stopping: 1/10 0.93495
stage 62/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92957 early_stopping: 2/10 0.93495
stage 63/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:59 val_accuracy: 0.92882 early_stopping: 3/10 0.93495
stage 64/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92863 early_stopping: 4/10 0.93495
stage 65/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93377 early_stopping: 5/10 0.93495
stage 66/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93218 early_stopping: 6/10 0.93495
stage 67/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93744 early_stopping: 0/10 0.93744
stage 68/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92683 early_stopping: 1/10 0.93744
stage 69/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93293 early_stopping: 2/10 0.93744
stage 70/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93520 early_stopping: 3/10 0.93744
stage 71/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93246 early_stopping: 4/10 0.93744
stage 72/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92340 early_stopping: 5/10 0.93744
stage 73/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:10 val_accuracy: 0.93548 early_stopping: 6/10 0.93744
stage 74/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:16 val_accuracy: 0.92944 early_stopping: 7/10 0.93744
stage 75/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93078 early_stopping: 8/10 0.93744
stage 76/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93094 early_stopping: 9/10 0.93744
stage 77/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93094 early_stopping: 9/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 77/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92941 early_stopping: 10/10 0.93744
stage 78/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92941 early_stopping: 10/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 78/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92044 early_stopping: 11/10 0.93744
stage 79/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92044 early_stopping: 11/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 79/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93489 early_stopping: 12/10 0.93744
stage 80/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93489 early_stopping: 12/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 80/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92359 early_stopping: 13/10 0.93744
stage 81/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93751 early_stopping: 0/10 0.93751
stage 82/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92848 early_stopping: 1/10 0.93751
stage 83/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93937 early_stopping: 0/10 0.93937
stage 84/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:00 val_accuracy: 0.92841 early_stopping: 1/10 0.93937
stage 85/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93716 early_stopping: 2/10 0.93937
stage 86/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92876 early_stopping: 3/10 0.93937
stage 87/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92412 early_stopping: 4/10 0.93937
stage 88/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93461 early_stopping: 5/10 0.93937
stage 89/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92627 early_stopping: 6/10 0.93937
stage 90/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93081 early_stopping: 7/10 0.93937
stage 91/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93542 early_stopping: 8/10 0.93937
stage 92/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93000 early_stopping: 9/10 0.93937
stage 93/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93000 early_stopping: 9/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 93/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93667 early_stopping: 10/10 0.93937
stage 94/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93667 early_stopping: 10/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 94/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93150 early_stopping: 11/10 0.93937
stage 95/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.93150 early_stopping: 11/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 95/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.93486 early_stopping: 12/10 0.93937
stage 96/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93486 early_stopping: 12/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 96/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93673 early_stopping: 13/10 0.93937
stage 97/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93673 early_stopping: 13/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 97/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93623 early_stopping: 14/10 0.93937
stage 98/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93623 early_stopping: 14/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 98/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.91889 early_stopping: 15/10 0.93937
stage 99/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.91889 early_stopping: 15/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 99/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93053 early_stopping: 16/10 0.93937
stage 100/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93053 early_stopping: 16/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 100/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93377 early_stopping: 17/10 0.93937
stage 101/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93377 early_stopping: 17/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 101/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93196 early_stopping: 18/10 0.93937
stage 102/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93196 early_stopping: 18/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 102/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93358 early_stopping: 19/10 0.93937
stage 103/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93358 early_stopping: 19/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 103/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93595 early_stopping: 20/10 0.93937
stage 104/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93595 early_stopping: 20/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 104/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92745 early_stopping: 21/10 0.93937
stage 105/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:45 val_accuracy: 0.92745 early_stopping: 21/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 105/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:45 val_accuracy: 0.93128 early_stopping: 22/10 0.93937
stage 106/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:50 val_accuracy: 0.93128 early_stopping: 22/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 106/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:50 val_accuracy: 0.92026 early_stopping: 23/10 0.93937
stage 107/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92026 early_stopping: 23/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 107/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93916 early_stopping: 24/10 0.93937
stage 108/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93916 early_stopping: 24/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 108/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93274 early_stopping: 25/10 0.93937
stage 109/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93274 early_stopping: 25/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 109/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92549 early_stopping: 26/10 0.93937
stage 110/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93635 early_stopping: 27/10 0.93937
stage 111/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93589 early_stopping: 28/10 0.93937
stage 112/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92695 early_stopping: 29/10 0.93937
stage 113/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93414 early_stopping: 30/10 0.93937
stage 114/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92760 early_stopping: 31/10 0.93937
stage 115/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93772 early_stopping: 32/10 0.93937
stage 116/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:10 val_accuracy: 0.93368 early_stopping: 33/10 0.93937
stage 117/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93947 early_stopping: 0/10 0.93947
stage 118/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93816 early_stopping: 1/10 0.93947
stage 119/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93732 early_stopping: 2/10 0.93947
stage 120/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92882 early_stopping: 3/10 0.93947
stage 121/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93084 early_stopping: 4/10 0.93947
stage 122/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93284 early_stopping: 5/10 0.93947
stage 123/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93271 early_stopping: 6/10 0.93947
stage 124/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93389 early_stopping: 7/10 0.93947
stage 125/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92788 early_stopping: 8/10 0.93947
stage 126/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.94028 early_stopping: 0/10 0.94028
stage 127/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:04 val_accuracy: 0.93582 early_stopping: 1/10 0.94028
stage 128/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92848 early_stopping: 2/10 0.94028
stage 129/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93598 early_stopping: 3/10 0.94028
stage 130/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92518 early_stopping: 4/10 0.94028
stage 131/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93461 early_stopping: 5/10 0.94028
stage 132/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92764 early_stopping: 6/10 0.94028
stage 133/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.94162 early_stopping: 0/10 0.94162
stage 134/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.94411 early_stopping: 0/10 0.94411
stage 135/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93835 early_stopping: 1/10 0.94411
stage 136/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93794 early_stopping: 2/10 0.94411
stage 137/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.94000 early_stopping: 3/10 0.94411
stage 138/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:57 val_accuracy: 0.93875 early_stopping: 4/10 0.94411
stage 139/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93735 early_stopping: 5/10 0.94411
stage 140/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93894 early_stopping: 6/10 0.94411
stage 141/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93327 early_stopping: 7/10 0.94411
stage 142/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93632 early_stopping: 8/10 0.94411
stage 143/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93729 early_stopping: 9/10 0.94411
stage 144/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92988 early_stopping: 10/10 0.94411
stage 145/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93181 early_stopping: 11/10 0.94411
stage 146/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93701 early_stopping: 12/10 0.94411
stage 147/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.94056 early_stopping: 13/10 0.94411
stage 148/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.94401 early_stopping: 14/10 0.94411
stage 149/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:35 val_accuracy: 0.94548 early_stopping: 0/10 0.94548
stage 150/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92340 early_stopping: 1/10 0.94548
stage 151/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93997 early_stopping: 2/10 0.94548
stage 152/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93903 early_stopping: 3/10 0.94548
stage 153/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93421 early_stopping: 4/10 0.94548
stage 154/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:31 val_accuracy: 0.93856 early_stopping: 5/10 0.94548
stage 155/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93860 early_stopping: 6/10 0.94548
stage 156/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92138 early_stopping: 7/10 0.94548
stage 157/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93825 early_stopping: 8/10 0.94548
stage 158/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93171 early_stopping: 9/10 0.94548
stage 159/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:59 val_accuracy: 0.93087 early_stopping: 10/10 0.94548
stage 160/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93695 early_stopping: 11/10 0.94548
stage 161/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93364 early_stopping: 12/10 0.94548
stage 162/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.94330 early_stopping: 13/10 0.94548
stage 163/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93249 early_stopping: 14/10 0.94548
stage 164/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.94252 early_stopping: 15/10 0.94548
stage 165/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93324 early_stopping: 16/10 0.94548
stage 166/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.94255 early_stopping: 17/10 0.94548
stage 167/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93212 early_stopping: 18/10 0.94548
stage 168/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93785 early_stopping: 19/10 0.94548
stage 169/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:32 val_accuracy: 0.93906 early_stopping: 20/10 0.94548
stage 170/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:08 val_accuracy: 0.93794 early_stopping: 21/10 0.94548
stage 171/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:33 val_accuracy: 0.93003 early_stopping: 22/10 0.94548
stage 172/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92888 early_stopping: 23/10 0.94548
stage 173/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93140 early_stopping: 24/10 0.94548
stage 174/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93941 early_stopping: 25/10 0.94548
stage 175/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93255 early_stopping: 26/10 0.94548
stage 176/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93190 early_stopping: 27/10 0.94548
stage 177/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93414 early_stopping: 28/10 0.94548
stage 178/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93682 early_stopping: 29/10 0.94548
stage 179/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93069 early_stopping: 30/10 0.94548
stage 180/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93327 early_stopping: 31/10 0.94548
stage 181/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:01 val_accuracy: 0.92605 early_stopping: 32/10 0.94548
stage 182/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93193 early_stopping: 33/10 0.94548
stage 183/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93760 early_stopping: 34/10 0.94548
stage 184/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93265 early_stopping: 35/10 0.94548
stage 185/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92767 early_stopping: 36/10 0.94548
stage 186/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93747 early_stopping: 37/10 0.94548
stage 187/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93909 early_stopping: 38/10 0.94548
stage 188/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93695 early_stopping: 39/10 0.94548
stage 189/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93844 early_stopping: 40/10 0.94548
stage 190/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93053 early_stopping: 41/10 0.94548
stage 191/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:16 val_accuracy: 0.94034 early_stopping: 42/10 0.94548
stage 192/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:58 val_accuracy: 0.93421 early_stopping: 43/10 0.94548
stage 193/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93455 early_stopping: 44/10 0.94548
stage 194/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93383 early_stopping: 45/10 0.94548
stage 195/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93090 early_stopping: 46/10 0.94548
stage 196/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.94563 early_stopping: 0/10 0.94563
stage 197/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93657 early_stopping: 1/10 0.94563
stage 198/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.94068 early_stopping: 2/10 0.94563
stage 199/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93483 early_stopping: 3/10 0.94563
stage 200/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93050 early_stopping: 4/10 0.94563
stage 201/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93187 early_stopping: 5/10 0.94563
stage 202/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:07:09 val_accuracy: 0.94731 early_stopping: 0/10 0.94731
stage 203/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93396 early_stopping: 1/10 0.94731
stage 204/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93642 early_stopping: 2/10 0.94731
stage 205/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93501 early_stopping: 3/10 0.94731
stage 206/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93224 early_stopping: 4/10 0.94731
stage 207/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:20 val_accuracy: 0.94221 early_stopping: 5/10 0.94731
stage 208/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93473 early_stopping: 6/10 0.94731
stage 209/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93726 early_stopping: 7/10 0.94731
stage 210/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.94062 early_stopping: 8/10 0.94731
stage 211/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93928 early_stopping: 9/10 0.94731
stage 212/โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93931 early_stopping: 10/10 0.94731
Moving best model Juristische_Konsilien_Tuebingen+_202.mlmodel (0.9473143219947815) to Juristische_Konsilien_Tuebingen+_best.mlmodel
real 1387m15.438s
user 19775m16.197s
sys 11007m8.085s
Run training with HTR+ like network specification
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+256 -d cuda:0 --augment --workers 24 -r 0.0001 -B 1 --min-epochs 200 --lag 20 -w 0 -s '[256,64,0,1 Cr4,2,8,4,2 Cr4,2,32,1,1 Mp4,2,4,2 Cr3,3,64,1,1 Mp1,2,1,2 S1(1x0)1,3 Lbx256 Do0.5 Lbx256 Do0.5 Lbx256 Do0.5 Cr255,1,85,1,1]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/27/22 17:05:44] WARNING alphabet mismatch: chars in training set only: {'ยฝ', '๊', 'โก', 'ร', 'โ', 'รป', ']', '[', 'ยบ', 'โ', 'X', 'โ', '๊ธ', '๊ฏ', '=', 'โ '} (not included in accuracy test during train.py:304
training)
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
โโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ โ Name โ Type โ Params โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 0 โ net โ MultiParamSequential โ 15.2 M โ
โ 1 โ net.C_0 โ ActConv2D โ 72 โ
โ 2 โ net.C_1 โ ActConv2D โ 2.1 K โ
โ 3 โ net.Mp_2 โ MaxPool โ 0 โ
โ 4 โ net.C_3 โ ActConv2D โ 18.5 K โ
โ 5 โ net.Mp_4 โ MaxPool โ 0 โ
โ 6 โ net.S_5 โ Reshape โ 0 โ
โ 7 โ net.L_6 โ TransposedSummarizingRNN โ 921 K โ
โ 8 โ net.Do_7 โ Dropout โ 0 โ
โ 9 โ net.L_8 โ TransposedSummarizingRNN โ 1.6 M โ
โ 10 โ net.Do_9 โ Dropout โ 0 โ
โ 11 โ net.L_10 โ TransposedSummarizingRNN โ 1.6 M โ
โ 12 โ net.Do_11 โ Dropout โ 0 โ
โ 13 โ net.C_12 โ ActConv2D โ 11.1 M โ
โ 14 โ net.O_13 โ LinSoftmax โ 10.4 K โ
โโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
Trainable params: 15.2 M
Non-trainable params: 0
Total params: 15.2 M
Total estimated model params size (MB): 60
stage 0/โ โโโโโโบโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 956/7366 0:05:12 0:00:58 early_stopping: 0/20 -inf