Here's the method, what worked, what didn't, and what I'd do differently.
4,098 labelled training lines, 1,374 test lines. Once I actually looked at the images, two very different populations showed up. About 75% are clean single-line register copies around 50 px tall. The other 25% are big damaged originals with stains, holes and 2 to 4 partial lines. Early models scored 0.892 on the clean ones and 0.826 on the damaged ones. Only 5 training rows were genuinely broken, so I dropped just those.
The metric mixes word and character errors:
score = 1 − 0.5 × (W / 12 + C / 55)
Plugging in my numbers at one point (W = 1.360, C = 2.112) gives exactly my public score of 0.9241: 0.057 lost to words, 0.019 to characters. One word error costs about 4.6 character errors, so the real game is whole-word exactness: capitals, punctuation, and the ^ superscript convention.
The most useful number of the whole competition: on 1,875 training lines where two independently trained models agreed exactly, the agreed text still scored only 0.945 against the labels. The leftovers were said vs Said, a missing ^, a comma, in both directions. That's annotator noise, so about 0.945 is the ceiling for everyone. It also killed pseudo-labelling for me, since agreed lines carry the same noise.
Grouped 5-fold CV, with near-duplicate transcriptions kept in the same fold. These documents repeat the same legal formulas constantly, so a random split leaks. My metric re-implementation lined up with Zindi from the first submission (CV 0.869 to 0.882, LB 0.8836).
The rule I ended up living by: only trust gains measured on the full 4,098-line OOF set, never on one fold. Adding a CTC model looked like +0.0030 on full OOF and gave +0.0033 on the LB. Adding Qwen3-VL-32B looked like +0.0032 on fold 0 alone and gave −0.0004. One fold is about 820 lines, so +0.003 there is roughly 59 words, which is just noise. Using all five folds cuts that noise by about √5 ≈ 2.2.
Several readers each write their n-best candidates for every line. Everything goes into one pool, and a weighted MBR vote picks the final line.
Everything was trained on 5 folds, and for the final submissions I also retrained the main models on all 4,098 lines. My last builds read every test image at three heights too (1.0, 0.9, 1.1).
The single biggest jump came from putting LoRA on the vision tower, not just the language layers: 0.869 to 0.905. With language-only LoRA the model wrote fluent nonsense, like "Powell" where the ink clearly says "Howlett". The language prior was overriding the image.
For each line, every candidate in the pool is scored by how far it sits from all the others, weighted, and the lowest-risk one wins:
best line = argmin over c of Σ_j w_j × 0.5 × (WER(c_j, c) + CER(c_j, c)) w_j = model_weight × exp(−0.4 × rank_j)
It uses the same word/char mix as the metric, so it's directly picking the line with the best expected score. Lower beam ranks get down-weighted (about 0.67 for the second guess, 0.45 for the third). I tuned the weights by coordinate search on the OOF pools. Caching the pairwise distances once per line turned that search from hours into seconds. By September the pool had 61 candidate sources.
If two models make the same mistake, no vote can fix it. TrOCR agreed with the rest of the pool far less often than the Qwens agreed with each other (about 28% vs 47%), and that's exactly why it earned its weight.
Talways picking the best candidate in the pool was 0.9548, while MBR landed at 0.924. That 0.031 gap looked like free points. It wasn't:
Most of that oracle gap is just the label noise again. The models disagree on said/Said and ^, exactly where the labels themselves are inconsistent.
Best public: 0.92940 (sub26, full-data models plus TTA), which scored 0.92401 on private. Best private: 0.92617, from sub19 back in early September, which I didn't select. It would've been around 26th.
Funny thing is, my own notes from September said: pick the best-CV entry plus one diversified entry, not the two best public scores. but i forgot about this competition deadline hahahah and that what happened
Brainiac's write-up (0.93306 private) is worth a read. A lot of it overlaps with what I found: grouped folds, LoRA on vision plus language, bigger models not helping, GRPO, rerankers and pseudo-labels not working, and a CTC-style model (Kraken) getting the most votes because it keeps the old spellings that VLMs modernise.
The big difference is that everything was word-aware:
The part that stings: I had ctc_align.py, a CTC forced aligner that does exactly that first step. I used it to cut words for StackMix and glue together fake sentences, which hurt the VLMs. Cutting real contiguous spans instead would have given them more real data in a real hand. Same tool, different idea.
The code will be on my GitHub soon: the metric, the folds, training and inference for Qwen3-VL, Qwen3.5, TrOCR and the CTC reader, the CTC aligner, StackMix, and the MBR ensemble that rebuilds a submission on CPU.
Thanks to Zindi and the organisers, everyone in the discussion threads, and Brainiac for the write-up this was one of the best competition on zindi history