MDiff4STR: Mask Diffusion Model for Scene Text Recognition

TL;DR

The paper introduces MDiff4STR, a Mask Diffusion Model tailored for Scene Text Recognition (STR), addressing the accuracy gap vs ARMs by mitigating training/inference noising gaps with six specialized training-noising strategies and a token-replacement noise mechanism. It leverages omnidirectional language modeling to outperform state-of-the-art ARMs while maintaining fast inference through only three denoising steps, and demonstrates robustness across regular, irregular, occluded, and Chinese text, with beneficial effects from pretraining. MDiff4STR supports multiple decoding paradigms (AR, PD, Re) and introduces LC/BLC strategies to improve denoising stability and error correction. Overall, the approach delivers strong, efficient STR performance and offers a new diffusion-based paradigm for scene text understanding with practical implications for multilingual and challenging text scenarios.

Abstract

Mask Diffusion Models (MDMs) have recently emerged as a promising alternative to auto-regressive models (ARMs) for vision-language tasks, owing to their flexible balance of efficiency and accuracy. In this paper, for the first time, we introduce MDMs into the Scene Text Recognition (STR) task. We show that vanilla MDM lags behind ARMs in terms of accuracy, although it improves recognition efficiency. To bridge this gap, we propose MDiff4STR, a Mask Diffusion model enhanced with two key improvement strategies tailored for STR. Specifically, we identify two key challenges in applying MDMs to STR: noising gap between training and inference, and overconfident predictions during inference. Both significantly hinder the performance of MDMs. To mitigate the first issue, we develop six noising strategies that better align training with inference behavior. For the second, we propose a token-replacement noise mechanism that provides a non-mask noise type, encouraging the model to reconsider and revise overly confident but incorrect predictions. We conduct extensive evaluations of MDiff4STR on both standard and challenging STR benchmarks, covering diverse scenarios including irregular, artistic, occluded, and Chinese text, as well as whether the use of pretraining. Across these settings, MDiff4STR consistently outperforms popular STR models, surpassing state-of-the-art ARMs in accuracy, while maintaining fast inference with only three denoising steps. Code: https://github.com/Topdu/OpenOCR.

Figures (6)

• Figure 1: (a) Auto-regressive models (ARMs), (b) Parallel decoding models (PDMs), (c) BERT-like refinement models (ReMs), (d) Mask diffusion models (MDMs). $\mathbf{F}_v$ means visual features. MDMs learn to reconstruct character sequences from partially masked inputs through a denoising process, capturing more flexible and comprehensive omnidirectional dependencies than ARMs and refinement models.
• Figure 2: The network of MDiff4STR. $\mathbf{F}_v$ and $\mathbf{T}_m$ denote the visual features and the noised tokens, respectively.
• Figure 3: (a) and (b) denote the denoising training and inference of the vanilla MDM, respectively. (c) depicts the error-correction training enabled by our proposed token-replacement noise mechanism. MDiff4STR jointly leverages (a) denoising and (c) error-correction training to achieve (d) the denoising process augmented with corrective capability. Red boxes indicate errors caused by overconfident predictions, whereas green boxes highlight correct reasoning performed by MDiff4STR. $\mathcal{M}_T$ and $\mathcal{M}_I$ denote the mask strategy for training and the remask strategy for inference, respectively.
• Figure 4: (a) illustrates random token masking for training. (b) shows the full mask strategy, which is used as the initial denoising step during inference. Subfigures (b–g) present various remasking strategies for inference and also server as noise strategies in training to eliminate the noising gap.
• Figure 5: The first two figures present the denoising process of MDiff4STR, while the last three demonstrate its reasoning advantage over ARM and ReM in omnidirectional contextual modeling involving occluded or artistic text recognition. V-MDiff4STR represents indicates that the token-replacement noise mechanism is not used during training. Red characters and underline denote the misrecognition and misspelling, respectively. Red boxes indicate errors caused by overconfident predictions, whereas green boxes highlight correct reasoning performed by MDiff4STR. * indicates tokens that are remasked as the [mask].