Training German Newspapers - UB-Mannheim/kraken GitHub Wiki

Training of kraken model for German newspapers

Ground Truth

Deutscher Reichsanzeiger
Ground truth for German newspaper "Deutscher Reichsanzeiger und Preußischer Staatsanzeiger" (1819–1945)
Dataset: https://github.com/UB-Mannheim/reichsanzeiger-gt

Austrian Newspapers
NewsEye / READ OCR training dataset from Austrian Newspapers (1864–1911)
Dataset: https://github.com/UB-Mannheim/AustrianNewspapers

Neue Zürcher Zeitung
Ground truth for swiss newspaper "Neue Zürcher Zeitung" (1780–1947)
Dataset: https://github.com/UB-Mannheim/NZZ-black-letter-ground-truth

(based on STRÖBEL, Phillip; CLEMATIDE, Simon. Improving OCR of black letter in historical newspapers: the unreasonable effectiveness of HTR models on low-resolution images. 2019.)

Hakenkreuzbanner
Ground truth for a political newspaper of the Mannheim region (1931–1945)
Dataset: https://github.com/UB-Mannheim/hkb-gt Evaluationset(!)

Preparing data for training

git clone https://github.com/UB-Mannheim/reichsanzeiger-gt
git clone https://github.com/UB-Mannheim/AustrianNewspapers
git clone https://github.com/UB-Mannheim/NZZ-black-letter-ground-truth
git clone https://github.com/UB-Mannheim/hkb-gt

./reichsanzeiger-gt/data/download_images.sh
mv -r ./reichsanzeiger-gt/data/images/* ./reichsanzeiger-gt/data/reichsanzeiger-1820-1939/GT-PAGE
./hkb-gt/data/download_images_to_page.sh

fdfind -a --full-path './reichsanzeiger-gt/data/reichsanzeiger-1820-1939/' -e xml >> german_newspapers.list
fdfind -a --full-path './NZZ-black-letter-ground-truth/data' -e xml >> german_newspapers.list 
fdfind -a --full-path './AustrianNewspapers/data' -e xml >> german_newspapers.list 
fdfind -a --full-path './hkb-gt/data' -e xml >> german_newspapers.list

shuf < german_newspapers.list | shuf > shuf_german_newspapers.list

export OMP_NUM_THREADS=1

Training with basemodel

Download basemodel german_print

wget https://ub-backup.bib.uni-mannheim.de/~stweil/tesstrain/kraken/german_print/german_print_best.mlmodel

Creating an apache arrow file with ketos -> german_newspapers.arrow is about 4 GB in size

time ketos compile --format-type xml --files ./shuf_german_newspapers.list  --workers 12 -o ./german_newspapers.arrow

Extracting lines ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 236003/236003 -:--:-- -:--:--
Output file written to ./german_newspapers.arrow
ketos compile --format-type xml --files ./shuf_german_newspapers.list  12 -o   25965,31s user 14011,24s system 1173% cpu 56:45,47 total

Comparing different neural network topolgies for training a basemodel

Here we test the topologies most commonly used in the Kraken (kraken, htru) and Transkribus (htr+) communities, a topology generated by OpenAI's GPT (gpt), and a unique variant derived from these topologies (sgd).

Interestingly, the results showed that there were no significant differences in training cycles or achieved accuracies among most of these topologies. This suggests that the choice of topology may be less critical than previously thought, as long as a basic level of complexity is maintained.

The larger topologies did not show a noticeable positive impact on performance (on our validation set). This may indicate that the training data (just printed Text, no handwritting) and the variations of characters and fonts may be too low in complexity to fully exploit the potential of these neural networks. Therefore, when training with larger datasets, it might be useful to choose a more complex topology, especially using mixed datasets (prints and manuscripts).

More detailed information about the specific network topologies, their parameters, and the accuracies achieved can be found in the attached tables. This provides a comprehensive overview and allows interested parties to review the results in detail and evaluate them for their own applications.

Information about the parameter and training

Data preparation

The datasets contains: Reichsanzeiger-gt, Austrian Newspapers and NZZ-Blackletter.

(kraken-venv)  jkamlah@notebook20  ~/Coding/models/german_newspapers/kraken  time ketos compile --format-type xml --files ./shuf_german_newspapers_31-12-2023.list  --workers 12 -o ./german_newspapers_2023_12.arrow                                                                          
Extracting lines ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 222610/222610 -:--:-- -:--:--
Output file written to ./german_newspapers_2023_12.arrow
ketos compile --format-type xml --files  --workers 12 -o   21375,98s user 34524,70s system 1569% cpu 59:21,94 total

kraken

This network topology is often used and recommended by Benjaming Kiessling, the main developer of Kraken.
It is rather small, but performs quite well in the evaluation.

(kraken-venv)  ✘ jkamlah@notebook20  ~/Coding/models/german_newspapers/kraken  time nice ketos train -f binary  -o ./20231231/kraken/german_newspapers -d cuda:0 --lag 10 -r 0.0001 -B 4 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]' german_newspapers_2023_12.arrow 
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA RTX A4000 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Name      ┃ Type                     ┃ Params ┃                 In sizes ┃                Out sizes ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ val_cer   │ CharErrorRate            │      0 │                        ? │                        ? │
│ 1  │ val_wer   │ WordErrorRate            │      0 │                        ? │                        ? │
│ 2  │ net       │ MultiParamSequential     │  4.1 M │  [[1, 1, 120, 400], '?'] │   [[1, 264, 1, 50], '?'] │
│ 3  │ net.C_0   │ ActConv2D                │  1.3 K │  [[1, 1, 120, 400], '?'] │ [[1, 32, 120, 400], '?'] │
│ 4  │ net.Do_1  │ Dropout                  │      0 │ [[1, 32, 120, 400], '?'] │ [[1, 32, 120, 400], '?'] │
│ 5  │ net.Mp_2  │ MaxPool                  │      0 │ [[1, 32, 120, 400], '?'] │  [[1, 32, 60, 200], '?'] │
│ 6  │ net.C_3   │ ActConv2D                │ 40.0 K │  [[1, 32, 60, 200], '?'] │  [[1, 32, 60, 200], '?'] │
│ 7  │ net.Do_4  │ Dropout                  │      0 │  [[1, 32, 60, 200], '?'] │  [[1, 32, 60, 200], '?'] │
│ 8  │ net.Mp_5  │ MaxPool                  │      0 │  [[1, 32, 60, 200], '?'] │  [[1, 32, 30, 100], '?'] │
│ 9  │ net.C_6   │ ActConv2D                │ 55.4 K │  [[1, 32, 30, 100], '?'] │  [[1, 64, 30, 100], '?'] │
│ 10 │ net.Do_7  │ Dropout                  │      0 │  [[1, 64, 30, 100], '?'] │  [[1, 64, 30, 100], '?'] │
│ 11 │ net.Mp_8  │ MaxPool                  │      0 │  [[1, 64, 30, 100], '?'] │   [[1, 64, 15, 50], '?'] │
│ 12 │ net.C_9   │ ActConv2D                │  110 K │   [[1, 64, 15, 50], '?'] │   [[1, 64, 15, 50], '?'] │
│ 13 │ net.Do_10 │ Dropout                  │      0 │   [[1, 64, 15, 50], '?'] │   [[1, 64, 15, 50], '?'] │
│ 14 │ net.S_11  │ Reshape                  │      0 │   [[1, 64, 15, 50], '?'] │   [[1, 960, 1, 50], '?'] │
│ 15 │ net.L_12  │ TransposedSummarizingRNN │  1.9 M │   [[1, 960, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 16 │ net.Do_13 │ Dropout                  │      0 │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 17 │ net.L_14  │ TransposedSummarizingRNN │  963 K │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 18 │ net.Do_15 │ Dropout                  │      0 │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 19 │ net.L_16  │ TransposedSummarizingRNN │  963 K │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 20 │ net.Do_17 │ Dropout                  │      0 │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 21 │ net.O_18  │ LinSoftmax               │  105 K │   [[1, 400, 1, 50], '?'] │   [[1, 264, 1, 50], '?'] │
└────┴───────────┴──────────────────────────┴────────┴──────────────────────────┴──────────────────────────┘
Trainable params: 4.1 M                                                                                                                                                                
Non-trainable params: 0                                                                                                                                                                
Total params: 4.1 M                                                                                                                                                                    
Total estimated model params size (MB): 16                                                                                                                                             
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:03 • 0:00:00 14.33it/s val_accuracy: 0.985 val_word_accuracy: 0.928  early_stopping: 0/10 0.98519
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:05 • 0:00:00 14.21it/s val_accuracy: 0.989 val_word_accuracy: 0.949  early_stopping: 0/10 0.98946
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:05 • 0:00:00 14.16it/s val_accuracy: 0.991 val_word_accuracy: 0.956  early_stopping: 0/10 0.99098
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:52 • 0:00:00 14.31it/s val_accuracy: 0.992 val_word_accuracy: 0.961  early_stopping: 0/10 0.99178
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.44it/s val_accuracy: 0.992 val_word_accuracy: 0.962  early_stopping: 0/10 0.99218
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.26it/s val_accuracy: 0.993 val_word_accuracy: 0.966  early_stopping: 0/10 0.99289
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.13it/s val_accuracy: 0.993 val_word_accuracy: 0.966  early_stopping: 0/10 0.99292
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.20it/s val_accuracy: 0.993 val_word_accuracy: 0.966  early_stopping: 1/10 0.99292
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:59 • 0:00:00 14.02it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 0/10 0.99344
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:54 • 0:00:00 14.13it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 1/10 0.99344
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:01 • 0:00:00 14.28it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 2/10 0.99344
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:02 • 0:00:00 14.25it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99362
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:03 • 0:00:00 14.16it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99363
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:00 • 0:00:00 14.30it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99371
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:56 • 0:00:00 14.11it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99380
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:59 • 0:00:00 14.20it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99385
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:54 • 0:00:00 14.08it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99390
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.18it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99390
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:56 • 0:00:00 14.34it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99390
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:02 • 0:00:00 13.84it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99395
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:52 • 0:00:00 14.17it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99401
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:54 • 0:00:00 14.28it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 1/10 0.99401
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:58 • 0:00:00 14.34it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99402
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:55 • 0:00:00 14.00it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99406
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:56 • 0:00:00 14.13it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99414
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.17it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 1/10 0.99414
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:03 • 0:00:00 14.15it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 2/10 0.99414
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:06 • 0:00:00 13.90it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99419
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:01 • 0:00:00 14.10it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 1/10 0.99419
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:51 • 0:00:00 14.54it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 2/10 0.99419
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.15it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 3/10 0.99419
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.12it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 4/10 0.99419
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:58 • 0:00:00 14.48it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 5/10 0.99419
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:58 • 0:00:00 14.24it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 6/10 0.99419
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:55 • 0:00:00 14.31it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 7/10 0.99419
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:57 • 0:00:00 14.18it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 8/10 0.99419
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:52 • 0:00:00 14.09it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 9/10 0.99419
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:55 • 0:00:00 13.97it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99427
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:03 • 0:00:00 14.23it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99431
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:00 • 0:00:00 14.23it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 1/10 0.99431
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:54 • 0:00:00 14.38it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 2/10 0.99431
stage 41/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:55 • 0:00:00 14.08it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 3/10 0.99431
stage 42/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:51 • 0:00:00 14.06it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 4/10 0.99431
stage 43/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:00:04 • 0:00:00 13.78it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 5/10 0.99431
stage 44/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:54 • 0:00:00 13.70it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 6/10 0.99431
stage 45/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:59 • 0:00:00 14.05it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 7/10 0.99431
stage 46/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:00:00 • 0:00:00 13.61it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 8/10 0.99431
stage 47/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:00:03 • 0:00:00 13.76it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 9/10 0.99431
stage 48/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:58 • 0:00:00 13.85it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 10/10 0.99431
Moving best model ./20231231/kraken/german_newspapers_38.mlmodel (0.9943079948425293) to ./20231231/kraken/german_newspapers_best.mlmodel
nice ketos train -f binary -o ./20231231/krakenD/german_newspapers -d cuda:0   210504,49s user 7813,78s system 119% cpu 50:42:36,20 total

htru

This network topology is often used by Thibault Clérice and Alix Chague, the main developer of HTR-United.
It is quite complex and could potentially outperform smaller networks if manuscripts or mixed datasets were used.

time nice ketos train -f binary  -o ./20231231/htru/german_newspapers -d cuda:0 --lag 10 -r 0.0001 -B 4 -w 0 -s '[1,120,0,1 Cr4,2,32,4,2 Gn32 Cr4,2,64,1,1 Gn32 Mp4,2,4,2 Cr3,3,128,1,1 Gn32 Mp1,2,1,2 S1(1x0)1,3 Lbx256 Do0.5 Lbx256 Do0.5 Lbx256 Do0.5]' german_newspapers_2023_12.arrow
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA RTX A4000 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Name      ┃ Type                     ┃ Params ┃                In sizes ┃               Out sizes ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ val_cer   │ CharErrorRate            │      0 │                       ? │                       ? │
│ 1  │ val_wer   │ WordErrorRate            │      0 │                       ? │                       ? │
│ 2  │ net       │ MultiParamSequential     │  5.7 M │ [[1, 1, 120, 400], '?'] │  [[1, 264, 1, 49], '?'] │
│ 3  │ net.C_0   │ ActConv2D                │    288 │ [[1, 1, 120, 400], '?'] │ [[1, 32, 30, 200], '?'] │
│ 4  │ net.Gn_1  │ GroupNorm                │     64 │ [[1, 32, 30, 200], '?'] │ [[1, 32, 30, 200], '?'] │
│ 5  │ net.C_2   │ ActConv2D                │ 16.4 K │ [[1, 32, 30, 200], '?'] │ [[1, 64, 29, 199], '?'] │
│ 6  │ net.Gn_3  │ GroupNorm                │    128 │ [[1, 64, 29, 199], '?'] │ [[1, 64, 29, 199], '?'] │
│ 7  │ net.Mp_4  │ MaxPool                  │      0 │ [[1, 64, 29, 199], '?'] │   [[1, 64, 7, 99], '?'] │
│ 8  │ net.C_5   │ ActConv2D                │ 73.9 K │   [[1, 64, 7, 99], '?'] │  [[1, 128, 7, 99], '?'] │
│ 9  │ net.Gn_6  │ GroupNorm                │    256 │  [[1, 128, 7, 99], '?'] │  [[1, 128, 7, 99], '?'] │
│ 10 │ net.Mp_7  │ MaxPool                  │      0 │  [[1, 128, 7, 99], '?'] │  [[1, 128, 7, 49], '?'] │
│ 11 │ net.S_8   │ Reshape                  │      0 │  [[1, 128, 7, 49], '?'] │  [[1, 896, 1, 49], '?'] │
│ 12 │ net.L_9   │ TransposedSummarizingRNN │  2.4 M │  [[1, 896, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 13 │ net.Do_10 │ Dropout                  │      0 │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 14 │ net.L_11  │ TransposedSummarizingRNN │  1.6 M │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 15 │ net.Do_12 │ Dropout                  │      0 │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 16 │ net.L_13  │ TransposedSummarizingRNN │  1.6 M │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 17 │ net.Do_14 │ Dropout                  │      0 │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 18 │ net.O_15  │ LinSoftmax               │  135 K │  [[1, 512, 1, 49], '?'] │  [[1, 264, 1, 49], '?'] │
└────┴───────────┴──────────────────────────┴────────┴─────────────────────────┴─────────────────────────┘
Trainable params: 5.7 M                                                                                                                                                                
Non-trainable params: 0                                                                                                                                                                
Total params: 5.7 M                                                                                                                                                                    
Total estimated model params size (MB): 22                                                                                                                                             
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:48:05 • 0:00:00 17.44it/s val_accuracy: 0.987 val_word_accuracy: 0.935  early_stopping: 0/10 0.98660
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:49:45 • 0:00:00 16.97it/s val_accuracy: 0.99 val_word_accuracy: 0.951  early_stopping: 0/10 0.98987
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:49:49 • 0:00:00 16.99it/s val_accuracy: 0.991 val_word_accuracy: 0.957  early_stopping: 0/10 0.99111
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:36 • 0:00:00 16.62it/s val_accuracy: 0.992 val_word_accuracy: 0.96  early_stopping: 0/10 0.99188
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:32 • 0:00:00 16.26it/s val_accuracy: 0.992 val_word_accuracy: 0.964  early_stopping: 0/10 0.99247
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:42 • 0:00:00 16.14it/s val_accuracy: 0.993 val_word_accuracy: 0.965  early_stopping: 0/10 0.99282
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:55:55 • 0:00:00 13.90it/s val_accuracy: 0.992 val_word_accuracy: 0.963  early_stopping: 1/10 0.99282
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:22 • 0:00:00 16.70it/s val_accuracy: 0.993 val_word_accuracy: 0.966  early_stopping: 0/10 0.99306
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:55:13 • 0:00:00 13.74it/s val_accuracy: 0.993 val_word_accuracy: 0.967  early_stopping: 0/10 0.99325
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:18 • 0:00:00 16.38it/s val_accuracy: 0.993 val_word_accuracy: 0.967  early_stopping: 1/10 0.99325
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:37 • 0:00:00 16.44it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 0/10 0.99350
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:28 • 0:00:00 16.18it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99383
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:08 • 0:00:00 16.45it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99383
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:30 • 0:00:00 16.49it/s val_accuracy: 0.994 val_word_accuracy: 0.969  early_stopping: 2/10 0.99383
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:32 • 0:00:00 16.42it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 3/10 0.99383
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:32 • 0:00:00 16.24it/s val_accuracy: 0.994 val_word_accuracy: 0.969  early_stopping: 4/10 0.99383
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:31 • 0:00:00 16.26it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99404
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:51 • 0:00:00 14.36it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99408
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:07 • 0:00:00 16.57it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 1/10 0.99408
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:35 • 0:00:00 16.37it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99420
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:38 • 0:00:00 14.41it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99424
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:23 • 0:00:00 14.08it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 1/10 0.99424
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:45 • 0:00:00 14.34it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99425
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:49 • 0:00:00 16.64it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 1/10 0.99425
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:37 • 0:00:00 16.34it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 2/10 0.99425
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:36 • 0:00:00 16.78it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99426
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:38 • 0:00:00 16.71it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99434
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:37 • 0:00:00 16.63it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 1/10 0.99434
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:38 • 0:00:00 16.63it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99438
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:22 • 0:00:00 16.68it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 1/10 0.99438
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:37 • 0:00:00 16.46it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 2/10 0.99438
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:41 • 0:00:00 16.21it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 3/10 0.99438
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:41 • 0:00:00 16.32it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 4/10 0.99438
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:37 • 0:00:00 16.66it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 5/10 0.99438
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:42 • 0:00:00 16.47it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99439
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:43 • 0:00:00 16.42it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99442
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:43 • 0:00:00 16.43it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99442
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:54:10 • 0:00:00 16.80it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 2/10 0.99442
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:29 • 0:00:00 16.31it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 3/10 0.99442
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:43 • 0:00:00 16.28it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 4/10 0.99442
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:44 • 0:00:00 16.29it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 5/10 0.99442
stage 41/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:21 • 0:00:00 16.64it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 6/10 0.99442
stage 42/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:53:59 • 0:00:00 16.45it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 7/10 0.99442
stage 43/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:29 • 0:00:00 16.48it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 8/10 0.99442
stage 44/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:41 • 0:00:00 16.55it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 9/10 0.99442
stage 45/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:44 • 0:00:00 16.68it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 10/10 0.99442
Moving best model ./20231231/htru/german_newspapers_35.mlmodel (0.9944181442260742) to ./20231231/htru/german_newspapers_best.mlmodel
nice ketos train -f binary -o ./20231231/htru/german_newspapers -d     183509,75s user 9714,60s system 128% cpu 41:43:41,61 total

htr+

This is the network topology was developed by CITlab and used in the Transkribus project.
It is quite complex and could potentially outperform smaller networks if manuscripts or mixed datasets were used.

time nice ketos train -f binary  -o ./20231231/htr+/german_newspapers -d cuda:0 --lag 10 -r 0.0001 -B 4 -w 0 -s '[1,128,0,1 Cr4,2,8,4,2 Cr4,2,32,1,1 Mp4,2,4,2 Cr3,3,64,1,1 Mp1,2,1,2 S1(1x0)1,3 Lbx256 Do0.5 Lbx256 Do0.5 Lbx256 Do0.5]' german_newspapers_2023_12.arrow
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA RTX A4000 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Name      ┃ Type                     ┃ Params ┃                In sizes ┃               Out sizes ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ val_cer   │ CharErrorRate            │      0 │                       ? │                       ? │
│ 1  │ val_wer   │ WordErrorRate            │      0 │                       ? │                       ? │
│ 2  │ net       │ MultiParamSequential     │  4.8 M │ [[1, 1, 128, 400], '?'] │  [[1, 264, 1, 49], '?'] │
│ 3  │ net.C_0   │ ActConv2D                │     72 │ [[1, 1, 128, 400], '?'] │  [[1, 8, 32, 200], '?'] │
│ 4  │ net.C_1   │ ActConv2D                │  2.1 K │  [[1, 8, 32, 200], '?'] │ [[1, 32, 31, 199], '?'] │
│ 5  │ net.Mp_2  │ MaxPool                  │      0 │ [[1, 32, 31, 199], '?'] │   [[1, 32, 7, 99], '?'] │
│ 6  │ net.C_3   │ ActConv2D                │ 18.5 K │   [[1, 32, 7, 99], '?'] │   [[1, 64, 7, 99], '?'] │
│ 7  │ net.Mp_4  │ MaxPool                  │      0 │   [[1, 64, 7, 99], '?'] │   [[1, 64, 7, 49], '?'] │
│ 8  │ net.S_5   │ Reshape                  │      0 │   [[1, 64, 7, 49], '?'] │  [[1, 448, 1, 49], '?'] │
│ 9  │ net.L_6   │ TransposedSummarizingRNN │  1.4 M │  [[1, 448, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 10 │ net.Do_7  │ Dropout                  │      0 │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 11 │ net.L_8   │ TransposedSummarizingRNN │  1.6 M │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 12 │ net.Do_9  │ Dropout                  │      0 │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 13 │ net.L_10  │ TransposedSummarizingRNN │  1.6 M │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 14 │ net.Do_11 │ Dropout                  │      0 │  [[1, 512, 1, 49], '?'] │  [[1, 512, 1, 49], '?'] │
│ 15 │ net.O_12  │ LinSoftmax               │  135 K │  [[1, 512, 1, 49], '?'] │  [[1, 264, 1, 49], '?'] │
└────┴───────────┴──────────────────────────┴────────┴─────────────────────────┴─────────────────────────┘
Trainable params: 4.8 M                                                                                                     
Non-trainable params: 0                                                                                                     
Total params: 4.8 M                                                                                                         
Total estimated model params size (MB): 19                                                                                  
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:17 • 0:00:00 18.54it/s val_accuracy: 0.983 early_stopping: 0/10   val_word_accuracy:   0.921   0.98301
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:46:34 • 0:00:00 17.92it/s val_accuracy: 0.988 early_stopping: 0/10   val_word_accuracy:   0.944   0.98826
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:48:04 • 0:00:00 17.12it/s val_accuracy: 0.991 early_stopping: 0/10   val_word_accuracy:   0.955   0.99067
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:48:17 • 0:00:00 17.64it/s val_accuracy: 0.992 early_stopping: 0/10   val_word_accuracy:   0.96    0.99153 
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:39 • 0:00:00 17.59it/s val_accuracy: 0.992 early_stopping: 0/10   val_word_accuracy:   0.962   0.99213
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:42 • 0:00:00 17.28it/s val_accuracy: 0.993 early_stopping: 0/10   val_word_accuracy:   0.964   0.99261 
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:39 • 0:00:00 17.36it/s val_accuracy: 0.993 early_stopping: 0/10   val_word_accuracy:   0.966   0.99285  
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:45 • 0:00:00 17.65it/s val_accuracy: 0.993 early_stopping: 0/10   val_word_accuracy:   0.967   0.99313 
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:46:58 • 0:00:00 18.58it/s val_accuracy: 0.993 early_stopping: 1/10   val_word_accuracy:   0.967   0.99313 
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:45 • 0:00:00 18.53it/s val_accuracy: 0.993 early_stopping: 0/10   val_word_accuracy:   0.968   0.99342 
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:43 • 0:00:00 18.07it/s val_accuracy: 0.993 early_stopping: 1/10   val_word_accuracy:   0.967   0.99342 
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:43 • 0:00:00 18.41it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.969   0.99369
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:46 • 0:00:00 18.34it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.97    0.99382
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:41 • 0:00:00 18.84it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.971   0.99389
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:45 • 0:00:00 18.38it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.971   0.99389 
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:45 • 0:00:00 18.49it/s val_accuracy: 0.994 early_stopping: 1/10   val_word_accuracy:   0.97    0.99389
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:42 • 0:00:00 18.49it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.972   0.99403 
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:45 • 0:00:00 18.31it/s val_accuracy: 0.993 early_stopping: 1/10   val_word_accuracy:   0.968   0.99403 
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:45 • 0:00:00 18.46it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.972   0.99410
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:46 • 0:00:00 18.41it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.972   0.99416 
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:47 • 0:00:00 18.18it/s val_accuracy: 0.994 early_stopping: 1/10   val_word_accuracy:   0.971   0.99416
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:46 • 0:00:00 18.49it/s val_accuracy: 0.994 early_stopping: 2/10   val_word_accuracy:   0.972   0.99416 
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:47 • 0:00:00 18.20it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.972   0.99419
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:49 • 0:00:00 18.49it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.973   0.99421
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:42 • 0:00:00 18.48it/s val_accuracy: 0.994 early_stopping: 0/10   val_word_accuracy:   0.973   0.99430
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:44 • 0:00:00 18.24it/s val_accuracy: 0.994 early_stopping: 1/10   val_word_accuracy:   0.973   0.99430 
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:46 • 0:00:00 18.01it/s val_accuracy: 0.994 early_stopping: 2/10   val_word_accuracy:   0.972   0.99430 
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:45:44 • 0:00:00 18.34it/s val_accuracy: 0.994 early_stopping: 3/10   val_word_accuracy:   0.972   0.99430 
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:18 • 0:00:00 17.95it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 4/10 0.99430
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:53:32 • 0:00:00 14.57it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 0/10 0.99438
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:54:49 • 0:00:00 15.01it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 1/10 0.99438
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:54:25 • 0:00:00 15.30it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 2/10 0.99438
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:48:15 • 0:00:00 17.84it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 3/10 0.99438
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:53:27 • 0:00:00 14.30it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 4/10 0.99438
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:49:38 • 0:00:00 17.53it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 5/10 0.99438
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:49:29 • 0:00:00 14.25it/s val_accuracy: 0.995 val_word_accuracy: 0.974  early_stopping: 0/10 0.99451
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:54:41 • 0:00:00 17.24it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 1/10 0.99451
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:52:41 • 0:00:00 17.80it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 2/10 0.99451
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:51:58 • 0:00:00 14.66it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 3/10 0.99451
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:51:29 • 0:00:00 17.50it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 4/10 0.99451
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:38 • 0:00:00 17.61it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 5/10 0.99451
stage 41/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:39 • 0:00:00 17.47it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 6/10 0.99451
stage 42/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:39 • 0:00:00 17.41it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 7/10 0.99451
stage 43/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:50:48 • 0:00:00 14.55it/s val_accuracy: 0.994 val_word_accuracy: 0.974  early_stopping: 8/10 0.99451
stage 44/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:49:31 • 0:00:00 17.31it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 9/10 0.99451
stage 45/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:47:42 • 0:00:00 17.41it/s val_accuracy: 0.994 val_word_accuracy: 0.973  early_stopping: 10/10 0.99451
Moving best model ./20231231/htr+/german_newspapers_35.mlmodel (0.9945129752159119) to ./20231231/htr+/german_newspapers_best.mlmodel
nice ketos train -f binary -o ./20231231/htr+/german_newspapers -d cuda:0  10  171697,09s user 11665,18s system 130% cpu 38:55:34,52 total

gpt

This is the network topology was recommended by ChatGPT based on the other networks as input.
It is quite complex and could potentially outperform smaller networks if manuscripts or mixed datasets were used.

time nice ketos train -f binary  -o ./20231231/gpt/german_newspapers -d cuda:0 --lag 10 -r 0.0001 -B 4 -w 0 -s '[1,120,0,1 Cr3,3,32,1,1 Gn32 Mp2,2 Cr3,3,64,1,1 Gn64 Mp2,2,2,2 Cr3,3,128,1,1 Gn128 Mp2,2,2,2 Cr3,3,256,1,1 Gn256 Mp2,2,2,2 S1(1x0)1,3 Lbx256 Do0.2 Lbx256 Do0.2 Lbx256 Do0.2]' german_newspapers_2023_12.arrow
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA RTX A4000 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Name      ┃ Type                     ┃ Params ┃                 In sizes ┃                Out sizes ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ val_cer   │ CharErrorRate            │      0 │                        ? │                        ? │
│ 1  │ val_wer   │ WordErrorRate            │      0 │                        ? │                        ? │
│ 2  │ net       │ MultiParamSequential     │  7.9 M │  [[1, 1, 120, 400], '?'] │   [[1, 264, 1, 25], '?'] │
│ 3  │ net.C_0   │ ActConv2D                │    320 │  [[1, 1, 120, 400], '?'] │ [[1, 32, 120, 400], '?'] │
│ 4  │ net.Gn_1  │ GroupNorm                │     64 │ [[1, 32, 120, 400], '?'] │ [[1, 32, 120, 400], '?'] │
│ 5  │ net.Mp_2  │ MaxPool                  │      0 │ [[1, 32, 120, 400], '?'] │  [[1, 32, 60, 200], '?'] │
│ 6  │ net.C_3   │ ActConv2D                │ 18.5 K │  [[1, 32, 60, 200], '?'] │  [[1, 64, 60, 200], '?'] │
│ 7  │ net.Gn_4  │ GroupNorm                │    128 │  [[1, 64, 60, 200], '?'] │  [[1, 64, 60, 200], '?'] │
│ 8  │ net.Mp_5  │ MaxPool                  │      0 │  [[1, 64, 60, 200], '?'] │  [[1, 64, 30, 100], '?'] │
│ 9  │ net.C_6   │ ActConv2D                │ 73.9 K │  [[1, 64, 30, 100], '?'] │ [[1, 128, 30, 100], '?'] │
│ 10 │ net.Gn_7  │ GroupNorm                │    256 │ [[1, 128, 30, 100], '?'] │ [[1, 128, 30, 100], '?'] │
│ 11 │ net.Mp_8  │ MaxPool                  │      0 │ [[1, 128, 30, 100], '?'] │  [[1, 128, 15, 50], '?'] │
│ 12 │ net.C_9   │ ActConv2D                │  295 K │  [[1, 128, 15, 50], '?'] │  [[1, 256, 15, 50], '?'] │
│ 13 │ net.Gn_10 │ GroupNorm                │    512 │  [[1, 256, 15, 50], '?'] │  [[1, 256, 15, 50], '?'] │
│ 14 │ net.Mp_11 │ MaxPool                  │      0 │  [[1, 256, 15, 50], '?'] │   [[1, 256, 7, 25], '?'] │
│ 15 │ net.S_12  │ Reshape                  │      0 │   [[1, 256, 7, 25], '?'] │  [[1, 1792, 1, 25], '?'] │
│ 16 │ net.L_13  │ TransposedSummarizingRNN │  4.2 M │  [[1, 1792, 1, 25], '?'] │   [[1, 512, 1, 25], '?'] │
│ 17 │ net.Do_14 │ Dropout                  │      0 │   [[1, 512, 1, 25], '?'] │   [[1, 512, 1, 25], '?'] │
│ 18 │ net.L_15  │ TransposedSummarizingRNN │  1.6 M │   [[1, 512, 1, 25], '?'] │   [[1, 512, 1, 25], '?'] │
│ 19 │ net.Do_16 │ Dropout                  │      0 │   [[1, 512, 1, 25], '?'] │   [[1, 512, 1, 25], '?'] │
│ 20 │ net.L_17  │ TransposedSummarizingRNN │  1.6 M │   [[1, 512, 1, 25], '?'] │   [[1, 512, 1, 25], '?'] │
│ 21 │ net.Do_18 │ Dropout                  │      0 │   [[1, 512, 1, 25], '?'] │   [[1, 512, 1, 25], '?'] │
│ 22 │ net.O_19  │ LinSoftmax               │  135 K │   [[1, 512, 1, 25], '?'] │   [[1, 264, 1, 25], '?'] │
└────┴───────────┴──────────────────────────┴────────┴──────────────────────────┴──────────────────────────┘
Trainable params: 7.9 M                                                                                                                                                                
Non-trainable params: 0                                                                                                                                                                
Total params: 7.9 M                                                                                                                                                                    
Total estimated model params size (MB): 31                                                                                                                                             
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:08 • 0:00:00 15.31it/s val_accuracy: 0.987 val_word_accuracy: 0.934  early_stopping: 0/10 0.98657
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:26 • 0:00:00 14.48it/s val_accuracy: 0.99 val_word_accuracy: 0.951  early_stopping: 0/10 0.98966
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:58 • 0:00:00 14.20it/s val_accuracy: 0.991 val_word_accuracy: 0.957  early_stopping: 0/10 0.99089
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:46 • 0:00:00 14.48it/s val_accuracy: 0.991 val_word_accuracy: 0.96  early_stopping: 0/10 0.99148
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:43 • 0:00:00 13.63it/s val_accuracy: 0.992 val_word_accuracy: 0.963  early_stopping: 0/10 0.99219
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:51 • 0:00:00 14.57it/s val_accuracy: 0.992 val_word_accuracy: 0.964  early_stopping: 0/10 0.99229
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:56 • 0:00:00 13.41it/s val_accuracy: 0.993 val_word_accuracy: 0.965  early_stopping: 0/10 0.99257
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:14 • 0:00:00 14.22it/s val_accuracy: 0.993 val_word_accuracy: 0.966  early_stopping: 0/10 0.99277
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:20 • 0:00:00 14.38it/s val_accuracy: 0.993 val_word_accuracy: 0.967  early_stopping: 0/10 0.99284
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:21 • 0:00:00 14.67it/s val_accuracy: 0.993 val_word_accuracy: 0.967  early_stopping: 0/10 0.99288
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:21 • 0:00:00 14.70it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 0/10 0.99317
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:36 • 0:00:00 13.09it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 1/10 0.99317
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:00:46 • 0:00:00 14.51it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 2/10 0.99317
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:53 • 0:00:00 14.80it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 0/10 0.99323
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:41 • 0:00:00 14.99it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 0/10 0.99326
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:41 • 0:00:00 14.80it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 1/10 0.99326
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:44 • 0:00:00 14.63it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 0/10 0.99327
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:42 • 0:00:00 14.58it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 0/10 0.99345
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:42 • 0:00:00 14.83it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 1/10 0.99345
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:43 • 0:00:00 14.85it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99353
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:46 • 0:00:00 14.61it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 1/10 0.99353
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:12 • 0:00:00 14.19it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 2/10 0.99353
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:22 • 0:00:00 14.09it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 3/10 0.99353
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:00:30 • 0:00:00 14.71it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99356
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:27 • 0:00:00 15.03it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 1/10 0.99356
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:42 • 0:00:00 14.42it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 2/10 0.99356
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:46 • 0:00:00 14.44it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 3/10 0.99356
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:01 • 0:00:00 14.50it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 4/10 0.99356
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:45 • 0:00:00 14.54it/s val_accuracy: 0.993 val_word_accuracy: 0.969  early_stopping: 5/10 0.99356
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:40 • 0:00:00 14.19it/s val_accuracy: 0.994 val_word_accuracy: 0.969  early_stopping: 6/10 0.99356
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:45 • 0:00:00 13.70it/s val_accuracy: 0.994 val_word_accuracy: 0.969  early_stopping: 7/10 0.99356
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:43 • 0:00:00 14.88it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 8/10 0.99356
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:29 • 0:00:00 14.86it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99357
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:00 • 0:00:00 13.28it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99358
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:43 • 0:00:00 15.01it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99360
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:25 • 0:00:00 14.71it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99360
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:10 • 0:00:00 14.92it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99360
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:06 • 0:00:00 14.31it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99364
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:20 • 0:00:00 13.65it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99364
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:10 • 0:00:00 14.97it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99364
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:05 • 0:00:00 14.74it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 3/10 0.99364
stage 41/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:58 • 0:00:00 14.84it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 4/10 0.99364
stage 42/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:26 • 0:00:00 13.70it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 5/10 0.99364
stage 43/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:31 • 0:00:00 14.74it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99369
stage 44/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:08 • 0:00:00 14.99it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99369
stage 45/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:00:15 • 0:00:00 14.65it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99369
stage 46/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:55 • 0:00:00 14.42it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99370
stage 47/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:27 • 0:00:00 14.38it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99370
stage 48/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:55 • 0:00:00 14.74it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99371
stage 49/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:27 • 0:00:00 14.39it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99371
stage 50/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:13 • 0:00:00 14.28it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99372
stage 51/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:59:01 • 0:00:00 14.41it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99372
stage 52/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:58:44 • 0:00:00 14.40it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99372
stage 53/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:38 • 0:00:00 14.47it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 3/10 0.99372
stage 54/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:31 • 0:00:00 14.33it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 4/10 0.99372
stage 55/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:33 • 0:00:00 14.31it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99380
stage 56/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:26 • 0:00:00 14.73it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99380
stage 57/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:57:31 • 0:00:00 14.40it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99380
stage 58/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:36 • 0:00:00 15.01it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 3/10 0.99380
stage 59/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:34 • 0:00:00 14.69it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99384
stage 60/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:37 • 0:00:00 14.88it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99384
stage 61/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:39 • 0:00:00 14.71it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 2/10 0.99384
stage 62/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:35 • 0:00:00 14.78it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 3/10 0.99384
stage 63/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:38 • 0:00:00 14.74it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 4/10 0.99384
stage 64/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:36 • 0:00:00 14.83it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 5/10 0.99384
stage 65/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:36 • 0:00:00 15.05it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 6/10 0.99384
stage 66/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:37 • 0:00:00 14.84it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 7/10 0.99384
stage 67/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:36 • 0:00:00 14.76it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 8/10 0.99384
stage 68/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:55 • 0:00:00 14.55it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 9/10 0.99384
stage 69/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 0:56:38 • 0:00:00 14.83it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 10/10 0.99384
Moving best model ./20231231/gpt/german_newspapers_56.mlmodel (0.9943616986274719) to ./20231231/gpt/german_newspapers_best.mlmodel

sdg

This is network topology was developed by Jan Kamlah in the OCR-D Project, based on the other networks as input.
It is quite complex and could potentially outperform smaller networks if manuscripts or mixed datasets were used.

time nice ketos train -f binary  -o ./20231231/sgd/german_newspapers -d cuda:0 --lag 10 -r 0.0001 -B 4 -w 0 -s '[1,144,0,1 Cr4,2,16,1,1 Mp4,2 Cr2,2,48,1,1, Gn24 Mp2,2 Cr2,2,72,1,1 Gn36 Mp2,2 S1(1x0)1,3 Lbx288 Do0.2,2 Lbx288 Do0.2,2 Lbx288]' german_newspapers_2023_12.arrow

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA RTX A4000 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Name      ┃ Type                     ┃ Params ┃                 In sizes ┃                Out sizes ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ val_cer   │ CharErrorRate            │      0 │                        ? │                        ? │
│ 1  │ val_wer   │ WordErrorRate            │      0 │                        ? │                        ? │
│ 2  │ net       │ MultiParamSequential     │  6.2 M │  [[1, 1, 144, 400], '?'] │   [[1, 264, 1, 49], '?'] │
│ 3  │ net.C_0   │ ActConv2D                │    144 │  [[1, 1, 144, 400], '?'] │ [[1, 16, 143, 399], '?'] │
│ 4  │ net.Mp_1  │ MaxPool                  │      0 │ [[1, 16, 143, 399], '?'] │  [[1, 16, 35, 199], '?'] │
│ 5  │ net.C_2   │ ActConv2D                │  3.1 K │  [[1, 16, 35, 199], '?'] │  [[1, 48, 34, 198], '?'] │
│ 6  │ net.Gn_3  │ GroupNorm                │     96 │  [[1, 48, 34, 198], '?'] │  [[1, 48, 34, 198], '?'] │
│ 7  │ net.Mp_4  │ MaxPool                  │      0 │  [[1, 48, 34, 198], '?'] │   [[1, 48, 17, 99], '?'] │
│ 8  │ net.C_5   │ ActConv2D                │ 13.9 K │   [[1, 48, 17, 99], '?'] │   [[1, 72, 16, 98], '?'] │
│ 9  │ net.Gn_6  │ GroupNorm                │    144 │   [[1, 72, 16, 98], '?'] │   [[1, 72, 16, 98], '?'] │
│ 10 │ net.Mp_7  │ MaxPool                  │      0 │   [[1, 72, 16, 98], '?'] │    [[1, 72, 8, 49], '?'] │
│ 11 │ net.S_8   │ Reshape                  │      0 │    [[1, 72, 8, 49], '?'] │   [[1, 576, 1, 49], '?'] │
│ 12 │ net.L_9   │ TransposedSummarizingRNN │  2.0 M │   [[1, 576, 1, 49], '?'] │   [[1, 576, 1, 49], '?'] │
│ 13 │ net.Do_10 │ Dropout                  │      0 │   [[1, 576, 1, 49], '?'] │   [[1, 576, 1, 49], '?'] │
│ 14 │ net.L_11  │ TransposedSummarizingRNN │  2.0 M │   [[1, 576, 1, 49], '?'] │   [[1, 576, 1, 49], '?'] │
│ 15 │ net.Do_12 │ Dropout                  │      0 │   [[1, 576, 1, 49], '?'] │   [[1, 576, 1, 49], '?'] │
│ 16 │ net.L_13  │ TransposedSummarizingRNN │  2.0 M │   [[1, 576, 1, 49], '?'] │   [[1, 576, 1, 49], '?'] │
│ 17 │ net.O_14  │ LinSoftmax               │  152 K │   [[1, 576, 1, 49], '?'] │   [[1, 264, 1, 49], '?'] │
└────┴───────────┴──────────────────────────┴────────┴──────────────────────────┴──────────────────────────┘
Trainable params: 6.2 M                                                                                                                                                                
Non-trainable params: 0                                                                                                                                                                
Total params: 6.2 M                                                                                                                                                                    
Total estimated model params size (MB): 24                                                                                                                                             
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:04:58 • 0:00:00 13.09it/s val_accuracy: 0.988 val_word_accuracy: 0.939  early_stopping: 0/10 0.98759
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:29 • 0:00:00 12.79it/s val_accuracy: 0.991 val_word_accuracy: 0.953  early_stopping: 0/10 0.99055
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:34 • 0:00:00 12.94it/s val_accuracy: 0.992 val_word_accuracy: 0.958  early_stopping: 0/10 0.99168
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:33 • 0:00:00 12.86it/s val_accuracy: 0.992 val_word_accuracy: 0.962  early_stopping: 0/10 0.99236
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:37 • 0:00:00 12.87it/s val_accuracy: 0.993 val_word_accuracy: 0.964  early_stopping: 0/10 0.99273
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:41 • 0:00:00 12.74it/s val_accuracy: 0.993 val_word_accuracy: 0.966  early_stopping: 0/10 0.99314
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:47 • 0:00:00 12.27it/s val_accuracy: 0.993 val_word_accuracy: 0.968  early_stopping: 0/10 0.99344
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:48 • 0:00:00 12.95it/s val_accuracy: 0.994 val_word_accuracy: 0.968  early_stopping: 0/10 0.99355
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:07:48 • 0:00:00 12.51it/s val_accuracy: 0.993 val_word_accuracy: 0.967  early_stopping: 1/10 0.99355
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:48 • 0:00:00 12.70it/s val_accuracy: 0.994 val_word_accuracy: 0.969  early_stopping: 0/10 0.99366
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:07:07 • 0:00:00 12.74it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99392
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:43 • 0:00:00 12.55it/s val_accuracy: 0.994 val_word_accuracy: 0.969  early_stopping: 1/10 0.99392
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:31 • 0:00:00 12.47it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99397
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:42 • 0:00:00 12.41it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99400
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:32 • 0:00:00 12.41it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 0/10 0.99403
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:19 • 0:00:00 12.77it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99403
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:27 • 0:00:00 12.84it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 2/10 0.99403
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:25 • 0:00:00 12.74it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 3/10 0.99403
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:26 • 0:00:00 12.89it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99414
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:29 • 0:00:00 12.79it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99414
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:34 • 0:00:00 12.92it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 2/10 0.99414
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:25 • 0:00:00 12.79it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 3/10 0.99414
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:26 • 0:00:00 12.94it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99416
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:26 • 0:00:00 12.60it/s val_accuracy: 0.994 val_word_accuracy: 0.97  early_stopping: 1/10 0.99416
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:31 • 0:00:00 12.81it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 2/10 0.99416
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:28 • 0:00:00 12.73it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 0/10 0.99426
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:27 • 0:00:00 12.62it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99426
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:31 • 0:00:00 12.73it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 2/10 0.99426
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:26 • 0:00:00 12.78it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99429
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:34 • 0:00:00 11.81it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99429
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:30 • 0:00:00 12.41it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 0/10 0.99436
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:07:38 • 0:00:00 12.34it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 1/10 0.99436
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:08:01 • 0:00:00 12.13it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 2/10 0.99436
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:07:04 • 0:00:00 12.69it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 3/10 0.99436
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:15 • 0:00:00 12.95it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 4/10 0.99436
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:07:30 • 0:00:00 11.97it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 5/10 0.99436
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:06 • 0:00:00 12.78it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 6/10 0.99436
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:40 • 0:00:00 12.85it/s val_accuracy: 0.994 val_word_accuracy: 0.972  early_stopping: 7/10 0.99436
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:06:42 • 0:00:00 12.65it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 8/10 0.99436
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:41 • 0:00:00 12.94it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 9/10 0.99436
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50088/50088 1:05:43 • 0:00:00 12.49it/s val_accuracy: 0.994 val_word_accuracy: 0.971  early_stopping: 10/10 0.99436
Moving best model ./20231231/sgd/german_newspapers_30.mlmodel (0.9943616986274719) to ./20231231/sgd/german_newspapers_best.mlmodel
nice ketos train -f binary -o ./20231231/sgd/german_newspapers -d cuda:0  199080,08s user 13389,96s system 123% cpu 47:37:04,78 total

Evaluation

The evaluation set contains pages from the newspaper Hakenkreuzbanner, which were not included in the training!
Evaluation also includes a newly trained Tesseract model (german_newspapers) and the current best Tesseract model for historical sources (frak2021).
All topologies are quite similar in their results and should be trained on a larger dataset to test the potential of more complex neural network structures.
For this or similar datasets, we recommend using the kraken topology as it is the smallest model and gives the best results.

📁 Top models for /home/jkamlah/Documents/projects/OCR-D/Models/Evaluation/german_print/hkb_1941-09-03_0005 
🥇   99.58  	Kraken_german_newspapers_default_deep
🥈   99.57  	Kraken_german_newspapers_htr+
🥉   99.57  	Kraken_german_newspapers_default
☁️   99.56  	Kraken_german_newspapers_sdg
☁️   99.56  	Kraken_german_newspapers_gpt
☁️   99.16  	Tesseract_german_newspapers
☁️   98.02  	Tesseract_frak2021
☁️   97.47  	Kraken_digi_tue
📁 Top models for /home/jkamlah/Documents/projects/OCR-D/Models/Evaluation/german_print/hkb_1931-01-03_0003 
🥇   99.77  	Kraken_german_newspapers_default
🥈   99.76  	Kraken_german_newspapers_default_deep
🥉   99.70  	Kraken_german_newspapers_gpt
☁️   99.68  	Kraken_german_newspapers_htr+
☁️   99.61  	Kraken_german_newspapers_sdg
☁️   99.50  	Kraken_digi_tue
☁️   99.12  	Tesseract_german_newspapers
☁️   98.58  	Tesseract_frak2021
📁 Top models for /home/jkamlah/Documents/projects/OCR-D/Models/Evaluation/german_print/hkb_1943-04-01_0008 
🥇   99.62  	Kraken_german_newspapers_htr+
🥈   99.55  	Kraken_german_newspapers_default
🥉   99.54  	Kraken_german_newspapers_gpt
☁️   99.52  	Kraken_german_newspapers_default_deep
☁️   99.48  	Kraken_german_newspapers_sdg
☁️   99.11  	Tesseract_german_newspapers
☁️   98.75  	Kraken_digi_tue
☁️   97.95  	Tesseract_frak2021
📁 Top models for /home/jkamlah/Documents/projects/OCR-D/Models/Evaluation/german_print/hkb_1931-01-03_0011 
🥇   100.00  	Kraken_german_newspapers_sdg
🥈   100.00  	Kraken_german_newspapers_default_deep
🥉   100.00  	Kraken_german_newspapers_default
☁️   99.80  	Kraken_german_newspapers_gpt
☁️   99.79  	Kraken_german_newspapers_htr+
☁️   99.52  	Kraken_digi_tue
☁️   99.41  	Tesseract_german_newspapers
☁️   99.12  	Tesseract_frak2021
📁 Top models for /home/jkamlah/Documents/projects/OCR-D/Models/Evaluation/german_print/hkb_1937-11-21_0026 
🥇   99.77  	Kraken_german_newspapers_sdg
🥈   99.77  	Kraken_german_newspapers_default
🥉   99.77  	Kraken_german_newspapers_gpt
☁️   99.73  	Kraken_german_newspapers_default_deep
☁️   99.68  	Kraken_german_newspapers_htr+
☁️   99.48  	Tesseract_german_newspapers
☁️   99.25  	Kraken_digi_tue
☁️   99.01  	Tesseract_frak2021
📁 Top models for /home/jkamlah/Documents/projects/OCR-D/Models/Evaluation/german_print/hkb_1945-01-01_0022 
🥇   99.01  	Kraken_german_newspapers_default_deep
🥈   99.00  	Kraken_german_newspapers_default
🥉   98.96  	Kraken_german_newspapers_htr+
☁️   98.80  	Kraken_german_newspapers_gpt
☁️   98.79  	Kraken_german_newspapers_sdg
☁️   97.63  	Kraken_digi_tue
☁️   96.20  	Tesseract_frak2021
☁️   96.20  	Tesseract_german_newspapers

🎽 Top models over all
🥇   99.61  	Kraken_german_newspapers_default
🥈   99.60  	Kraken_german_newspapers_default_deep
🥉   99.55  	Kraken_german_newspapers_htr+
☁️   99.54  	Kraken_german_newspapers_sdg
☁️   99.53  	Kraken_german_newspapers_gpt
☁️   98.75  	Tesseract_german_newspapers
☁️   98.69  	Kraken_digi_tue
☁️   98.15  	Tesseract_frak2021