Thanks to the organizers and Zindi for this amazing, exciting challenge.
At A Glance
Eight reader variants read every test line.
- They come from five base models, and each variant has five fold versions: 8 models × 5 folds (OOF).
- Some see the whole line alone; others also see closer views of its words.
Word positions serve two purposes.
- They let me create extra labelled training images and choose windows for the nine-view readers.
- Training transcripts provide the positions for training lines; separate models predict them for test lines.
A weighted vote combines the readings into one transcript.
- Kraken helps preserve old spellings that the vision-language models sometimes modernise.
Pipeline at a glance
Three parts of the pipeline use word positions:
-
Training alignment: matches the known transcript to positions in its image to prepare training data.
-
Extra training crops: cuts out one to five whole words and pairs them with the matching transcript substring, giving the readers more labelled examples.
-
Nine-view input: shows the whole line, six overlapping windows (sections of the line that start and end between words) and two filtered whole-line copies, during both training and prediction.
The seven steps below explain how these parts fit together.
1 · Five folds
The 4,098 labelled lines are split into five folds of 820, 820, 820, 819 and 819 lines.
How the split was made
- Lines with matching text, and images that are near-copies of each other, stay together in the same fold.
- I tried eight random splits that kept these groups together and balanced the kinds of lines in each fold.
- I kept the split whose folds were most similar in text and image statistics.
-
preprocessing/folds.py reproduces the split from the training images and transcripts, using configs/recipes/folds.yaml. Training reads the saved assignments in artifacts/folds/.
How it is used
- Each reader variant is trained five times. Each version trains on four folds and predicts the fifth.
- Combining these held-out predictions gives one out-of-fold (OOF) prediction for every training line.
- The cross-validation (CV) scores below use all 4,098 predictions.
How I chose models and voting weights
- I used the combined score across all five folds.
- Settings that helped fold 0 did not always help the others. Kraken looked unhelpful on fold 0 before proving valuable across all five folds.
- The public leaderboard, which covers only part of the test set, was a secondary check.
2 · Aligner: find every word in the training lines
The aligner answers: where does each word in the supplied transcript appear in the image?
The reader
- A ConvNeXt-large CTC reader trained on the other four folds.
- CTC reads a sequence of characters from an image without needing a separate box around every character.
How alignment works
- The transcript is already known. The algorithm finds where its characters fit the reader's image predictions.
- Each word boundary goes in the space between neighbouring words.
- This is called forced alignment: it locates the supplied words without rewriting the transcript.
What the positions are used for
- Cutting extra training crops (step 3).
- Teaching the test-time boundary models (step 4).
- Choosing the windows shown to nine-view readers during training (step 5).
3 · Word-span crops: more training images
What they are
- Smaller images containing one to five whole words, cut at the aligner's word positions.
- Each crop keeps the full line height and uses the exact matching substring of the original transcript as its label.
- Each reader uses crops only from its own four training folds.
Seven crop types
- Carets (^) and punctuation.
- Capitalisation differences.
- Similar spellings, such as a final -e.
- Numbers and Latin.
- Line endings.
- Random spans.
- Five-word spans.
Who trains with them
- Qwen3-VL, LightOnOCR-2 and Kraken, alongside the original lines.
Which images can supply extra crops
- Training lines on which the aligner accepts every word boundary; held-out and test lines never do.
- The ConvNeXt aligner supplies the crop boundaries.
4 · Word-boundary models: find the words in test lines
Test lines have no supplied transcript, so they need a different way to locate words. Seven ConvNeXt-large models estimate word positions from the image alone:
Two word counters
- Estimate the most likely word count from the CTC reader's character probabilities.
- One handles normal lines at 128 px high; the other handles tall scans at 192 px.
Five boundary finders
- One per fold, predicting how likely each position is to be a gap between words.
- They produce a score for every 4-px column of the resized line and combine their predictions.
From scores to boundaries
- The estimated word count determines how many gaps to look for: one fewer gap than words.
- Keep the strongest gap predictions at least 24 px apart in the resized image.
- Use fewer gaps if there are not enough usable predictions.
Who uses the boundaries
- Nine-view readers use them to choose the windows of the line to see.
- Single-strip readers use the whole image and need no word boundaries.
5 · Eight reader variants
Five base models → eight reader variants
- Qwen3-VL, Qwen3.5 and LightOnOCR-2 each have a single-strip and a nine-view variant.
- ConvNeXt and Kraken each have one single-strip variant.
Five fold versions each: 8 models × 5 folds (OOF)
- Each variant has five fold versions, one per held-out fold.
- Training targets are the original transcripts, with their spelling and formatting unchanged.
Single strip vs nine views
Single strip
- The whole line as one image.
Nine views: 1 + 6 + 2
- One whole-line image.
- Six overlapping windows that start and end at word boundaries.
- Two filtered copies of the whole line.
How the views work together
- The reader produces one transcript from all nine views of the same line.
- The six windows provide closer views; the whole-line images preserve context.
- Training lines use the aligner's word positions; test lines use the boundary models from step 4.
- The extra crops in step 3 are separate labelled training examples.
- On its own, a nine-view model scores a little higher than its single-strip twin.
- It also gets different words right, especially on large, faded scans. Combining both variants helps the vote.
How the models were trained
Training methods
- LoRA trains small adapter weights inside a model.
- A full fine-tune updates all of the model's weights.
Recipe
- Vision-language models: 2 epochs, effective batch 8, a cosine schedule that ends exactly at the last update.
- Qwen: LoRA at 2e-4 (mergers 1e-5), using Unsloth for Qwen3-VL and Transformers/PEFT for Qwen3.5. LightOnOCR-2: full fine-tune at 2e-5.
- CTC models: ConvNeXt 30 epochs, Kraken 12 epochs.
- Cost per fold on one RTX A6000: from about 1 hour (ConvNeXt) to about 21 hours (Qwen3-VL nine views).
Vision LoRA and mergers
- The Qwen variants train LoRA adapters in both the vision tower (the image-reading component) and the language model.
- The mergers, which hand image features to the language model, are trained in full.
- Training the vision side mattered most; the mergers added a little more.
Small models, fast loops
- With nine views, Qwen3-VL 8B matched Qwen3-VL 32B at a fraction of the training time.
- Much bigger models did not win.
- I compared learning rates, dropout settings, LoRA ranks and training schedules on the 8B model, four runs at a time.
Repeatable runs
- Fixed random seeds, deterministic GPU operations, TF32 off and a fixed order of training examples.
- Two runs of the same recipe come out byte-identical, even on two different cards of the same model.
- This makes it easier to measure the effect of changing a training setting.
Decoding
- Vision-language readers use beam search with 4 possible transcriptions in progress at each decoding step.
- If a vision-language reading reaches its length limit and repeats its ending, the repeated text is reduced to one copy. A line that is still unfinished is read again using the most likely next token at each step (greedy decoding).
- ConvNeXt uses greedy CTC decoding.
- Kraken uses beam 64 and keeps its 25 best readings.
6 · The weighted vote
The eight reader variants receive fixed voting weights that add up to 47. A larger weight gives a reader more influence over the final transcript.
Votes per model
- Qwen3-VL 5 + 5, Qwen3.5 5 + 5, LightOnOCR-2 6 + 6, ConvNeXt 3, Kraken 12.
- Kraken's 12 votes are shared across its 25 best readings.
- I chose these weights using held-out predictions, measuring how much each reader improves the combined result. Kraken adds the most and receives the largest weight.
Three steps, in the spirit of ROVER
1. Build possible answers. Align the readings character by character and vote on the differences. Repeat this with each reading as the starting point to create several candidate transcripts.
2. Choose one answer. Compare each candidate with all the readings, using their voting weights. Choose the candidate with the lowest weighted average distance. This distance combines word edits divided by 12 and character edits divided by 55, with equal weight, based on the competition formula:
score = 0.5 × (1 − W / 12) + 0.5 × (1 − C / 55)
In the competition score, W and C are average word and character edits per line, weighted by the square root of the reference word and character counts, respectively. Candidate selection uses the same edit costs with the readers' voting weights; it does not compute the full competition score. One word edit therefore costs as much as about 4.6 character edits in the candidate comparison.
3. Check individual words. A final pass can replace a word with an alternative supported by readers from at least two different model families.
How the final edits are made
- Fixed rules choose edits from the readers' predictions (ensemble/rewrite.py).
- The ensemble also applies the whitespace rules in configs/ensemble.yaml.
- It calls no additional language model or dictionary.
Why one word matters
- Saving one word on every line adds about 0.04 to the score.
- One word in every ten lines still adds about 0.004.
Why Kraken improves the combined prediction
- Vision-language models sometimes replace old spellings with modern ones.
- Kraken, a small CTC reader, is better at preserving the written letters in these cases.
- When the vision-language models split evenly on a word, Kraken is right more often than their most common answer.
- It has the most votes, and leaving it out hurts the vote more than leaving out any other model.
KenLM inside Kraken
What it is
- A character 6-gram KenLM: a language model that scores sequences of up to six characters. It is built only from the training transcripts, excluding the held-out fold when producing out-of-fold predictions.
Where it sits
- Inside Kraken's beam search: beam 64, LM weight 0.8, length bonus 2.
- It steers Kraken toward letter sequences that really occur in these documents, old spellings included.
Where it did not help
- In a vision-language model, and in re-ranking whole lines: both got worse.
- In a second CTC model: it added nothing to the vote.
- One language model, in one CTC reader, was enough.
7 · Submission
- All five fold versions of each reader variant read every test line.
- They share their variant's voting weight equally.
-
8 models × 5 folds, plus Kraken's alternative readings, contribute to the final prediction.
- The chosen transcript for each of the 1,374 test lines is written to final_submission.csv.
Exploratory experiments outside the submitted solution
These experiments were not adopted in the submitted solution. The list records excluded approaches; it is not a description of the submitted training runs. The supplied trainers use labelled competition lines and their derived crops; they do not train on test predictions.
-
Bigger base models. Qwen3-VL 32B, InternVL3.5-38B and Gemma-4 31B were slower and no better.
-
Windows that ignore word boundaries, and "zoom everything" views with much larger images.
-
Reinforcement learning on the score (GRPO). The reward went up while held-out scores went down.
-
Learned re-rankers on top of the vote.
-
Pseudo-labels. Students trained on the vote's own test readings mostly copied their teacher.
-
Rare-word specialists. They "fixed" old spellings that were already right.
-
Prompt rewrites after fine-tuning, word-level language models and dictionary snapping.
huge thanks
Thank you @__yassine__
Congratulations @Brainiac 👏👏 and thanks to the detailed solution description!
Thank you @Muhamed_Tuo
You have taste @Brainiac. Congrats 👏👏👏
Thank u @iam-dante. I appreciate it 😂
Congratulations @Brainiac.
This is very helpful. Thank you.
A couple of questions if I may:
How did you find the number of votes for the combination?
Did you try any augmentation in FT/Lora?
(if I may) what hardware did you use?
Thanks for a very well thought out and described solution.
Thanks for the kind words @fganong
1. How did I set the votes?
2. Augmentation in FT/LoRA? Yes, but only light augmentation:
3. Hardware
What an amazing write up @Brainiac and a well deserved win. My question is did you use any AI assisted research and coding and what was it like?
Thanks @Koleshjr! Yes, I used AI assistants, mainly for coding and bouncing ideas around. Big speed-up, but they need close supervision.
That was very kind of you. Thank you for your contribution. It really helped me see things that I had been racking my brain to figure out. You are, without a doubt, a true Brainiac.
Thank you @vivendolavida