Transformer-based handwritten text recognition (HTR) systems decode text in a single left-to-right pass while keeping visual encoder representations fixed, preventing later contextual evidence from resolving earlier visual ambiguities. We propose Semantic Flow Refinement (SFR), a lightweight two-pass decoding framework that interprets decoder hidden states from an initial pass as a semantic flow and uses them to refine visual tokens via cross-attention before final decoding. Building on a TrOCR baseline augmented with stroke-direction supervision, SFR introduces no additional training data or encoder modifications. On a low-legibility student handwriting benchmark, SFR achieves CER = 1.19% with TrOCR-Base – surpassing the previous best result of 1.69% CER obtained with TrOCR-Large – demonstrating that contextual refinement can substitute for model scale in challenging HTR settings.
You can read the full article here (paywalled): Springer