Primary competition visual

R.O.A.D. Barbados Historic Handwriting Challenge

Helping Barbados
$25 000 USD
Completed (~1 month ago)
Optical Character Recognition
Natural Language Processing
1900 joined
653 active
Starti
03 Jul 26
Closei
04 Oct 26
Reveali
05 Oct 26
User avatar
Higher School of Statistics and Information Analysis
My Road Barbados Challenge Journey
20 Oct 2026, 09:32 Β· 0

Here's the method, what worked, what didn't, and what I'd do differently.

4,098 labelled training lines, 1,374 test lines. Once I actually looked at the images, two very different populations showed up. About 75% are clean single-line register copies around 50 px tall. The other 25% are big damaged originals with stains, holes and 2 to 4 partial lines. Early models scored 0.892 on the clean ones and 0.826 on the damaged ones. Only 5 training rows were genuinely broken, so I dropped just those.

The metric mixes word and character errors:

score = 1 − 0.5 × (W / 12 + C / 55)

Plugging in my numbers at one point (W = 1.360, C = 2.112) gives exactly my public score of 0.9241: 0.057 lost to words, 0.019 to characters. One word error costs about 4.6 character errors, so the real game is whole-word exactness: capitals, punctuation, and the ^ superscript convention.

The most useful number of the whole competition: on 1,875 training lines where two independently trained models agreed exactly, the agreed text still scored only 0.945 against the labels. The leftovers were said vs Said, a missing ^, a comma, in both directions. That's annotator noise, so about 0.945 is the ceiling for everyone. It also killed pseudo-labelling for me, since agreed lines carry the same noise.

Validation

Grouped 5-fold CV, with near-duplicate transcriptions kept in the same fold. These documents repeat the same legal formulas constantly, so a random split leaks. My metric re-implementation lined up with Zindi from the first submission (CV 0.869 to 0.882, LB 0.8836).

The rule I ended up living by: only trust gains measured on the full 4,098-line OOF set, never on one fold. Adding a CTC model looked like +0.0030 on full OOF and gave +0.0033 on the LB. Adding Qwen3-VL-32B looked like +0.0032 on fold 0 alone and gave −0.0004. One fold is about 820 lines, so +0.003 there is roughly 59 words, which is just noise. Using all five folds cuts that noise by about √5 ≈ 2.2.

The pipeline

Several readers each write their n-best candidates for every line. Everything goes into one pool, and a weighted MBR vote picks the final line.

The readers

  • Qwen3-VL-8B with LoRA on both the vision tower and the language model: 0.905 OOF, the backbone.
  • Qwen3.5-9B: 0.870 to 0.914 depending on the fold, a second family with different errors.
  • TrOCR-large with wide (192×1536) and square (384×384) inputs: 0.878 and 0.850.
  • A CTC reader: 0.816 alone, and somehow the most valuable member (more on that below).
  • Qwen3-VL-32B: 0.905, same as the 8B. Bigger wasn't better.

Everything was trained on 5 folds, and for the final submissions I also retrained the main models on all 4,098 lines. My last builds read every test image at three heights too (1.0, 0.9, 1.1).

The single biggest jump came from putting LoRA on the vision tower, not just the language layers: 0.869 to 0.905. With language-only LoRA the model wrote fluent nonsense, like "Powell" where the ink clearly says "Howlett". The language prior was overriding the image.

The vote: MBR

For each line, every candidate in the pool is scored by how far it sits from all the others, weighted, and the lowest-risk one wins:

best line = argmin over c of  Σ_j  w_j × 0.5 × (WER(c_j, c) + CER(c_j, c))

w_j = model_weight × exp(−0.4 × rank_j)

It uses the same word/char mix as the metric, so it's directly picking the line with the best expected score. Lower beam ranks get down-weighted (about 0.67 for the second guess, 0.45 for the third). I tuned the weights by coordinate search on the OOF pools. Caching the pairwise distances once per line turned that search from hours into seconds. By September the pool had 61 candidate sources.

Decorrelation beats accuracy

  • The CTC reader (0.816 alone) gave my single biggest LB jump: +0.0033.
  • A correction model, which re-reads the image with the pool's candidates in the prompt, was my best single model at 0.909. It made the ensemble worse.
  • A CTC trained with extra StackMix synthetic data was +0.010 better on every fold. Swapping it in lost 0.0007; adding it next to the old one gained 0.00075.
  • GOT-OCR2.0 brought very different errors but sat at 0.67 alone and hurt at every weight. There's a floor, roughly 0.80 here.
  • More Qwen variants per fold gave +0.0002, then +0.0000. A new model family gave +0.0011.

If two models make the same mistake, no vote can fix it. TrOCR agreed with the rest of the pool far less often than the Qwens agreed with each other (about 28% vs 47%), and that's exactly why it earned its weight.

What didn't work

Talways picking the best candidate in the pool was 0.9548, while MBR landed at 0.924. That 0.031 gap looked like free points. It wasn't:

  • Learned reranker (23 features, gradient boosting): 0.9147 vs 0.9169 for plain MBR.
  • n-gram LM rescoring built from the training labels: the optimal weight was zero.
  • Rescoring by the fine-tuned model's own likelihood: 0.9003.
  • ROVER-style word-level voting: 0.9224 to 0.9233 vs 0.9240 for MBR. Errors are correlated along a line. When a model misreads a smudge it misreads the neighbours too, so splicing words from different readers builds a line nobody actually read.
  • GRPO with the metric as reward: +0.0007 for 4.5 GPU-hours. Cancelled.
  • A specialist model for the big damaged crops: 0.827 vs 0.846 for the general model.
  • StackMix synthetic lines for the VLMs: −0.0104. The n-gram-sampled text is ungrammatical and messes with the language prior. It was fine for CTC, which has no prior.

Most of that oracle gap is just the label noise again. The models disagree on said/Said and ^, exactly where the labels themselves are inconsistent.

Results

Best public: 0.92940 (sub26, full-data models plus TTA), which scored 0.92401 on private. Best private: 0.92617, from sub19 back in early September, which I didn't select. It would've been around 26th.

Funny thing is, my own notes from September said: pick the best-CV entry plus one diversified entry, not the two best public scores. but i forgot about this competition deadline hahahah and that what happened

What the winner did differently

Brainiac's write-up (0.93306 private) is worth a read. A lot of it overlaps with what I found: grouped folds, LoRA on vision plus language, bigger models not helping, GRPO, rerankers and pseudo-labels not working, and a CTC-style model (Kraken) getting the most votes because it keeps the old spellings that VLMs modernise.

The big difference is that everything was word-aware:

  • A CTC model force-aligns the known transcript to find every word in the training lines.
  • Those positions are used to cut real 1-to-5-word spans as extra training crops, labelled with the exact substring.
  • The readers get nine views of each line: the full line, six windows that start and end between words, and two filtered copies.
  • The language model (a character 6-gram KenLM) lives inside Kraken's CTC beam search, not in line-level rescoring.
  • The vote builds ROVER-style candidates, picks one with a weighted whole-line distance, then runs a word-level check that needs two model families to agree.

The part that stings: I had ctc_align.py, a CTC forced aligner that does exactly that first step. I used it to cut words for StackMix and glue together fake sentences, which hurt the VLMs. Cutting real contiguous spans instead would have given them more real data in a real hand. Same tool, different idea.

Takeaways

  • Estimate the label-noise ceiling early. It tells you which ideas can even matter.
  • Trust full OOF, not one fold, and definitely not the public LB.
  • Judge a model by its ensemble delta, not its standalone score, but it still has to clear a minimum.
  • Read the raw outputs. A −0.69 model hid in my ensemble for days.
  • Real augmented data before synthetic data.
  • Check disk space before long jobs. 19 GB of synthetic images cost me 9 GPU-hours of TrOCR training.

Code

The code will be on my GitHub soon: the metric, the folds, training and inference for Qwen3-VL, Qwen3.5, TrOCR and the CTC reader, the CTC aligner, StackMix, and the MBR ensemble that rebuilds a submission on CPU.

Thanks to Zindi and the organisers, everyone in the discussion threads, and Brainiac for the write-up this was one of the best competition on zindi history

Discussion 0 answers