Joint Low-level and High-level Textual Representation Learning with Multiple Masking Strategies

TL;DR

The paper addresses the domain gap between synthetic and real scene text data by introducing Multi-Masking Strategy (MMS), a self-supervised masked image modeling framework that jointly leverages random patch, block, and span masking to learn both low-level stroke and high-level contextual representations. MMS employs a shared ViT encoder with three branches and a pixel-wise reconstruction objective, enabling effective pre-training on unlabeled real data and downstream fine-tuning on labeled data. Empirical evaluations across text recognition, segmentation, and text-image super-resolution show that MMS outperforms state-of-the-art self-supervised methods, with strong data-efficiency (notably at low labeling ratios) and robust performance on curved and occluded text. The results demonstrate the practical impact of integrating diverse masking strategies to capture comprehensive textual representations for real-world OCR tasks.

Abstract

Most existing text recognition methods are trained on large-scale synthetic datasets due to the scarcity of labeled real-world datasets. Synthetic images, however, cannot faithfully reproduce real-world scenarios, such as uneven illumination, irregular layout, occlusion, and degradation, resulting in performance disparities when handling complex real-world images. Recent self-supervised learning techniques, notably contrastive learning and masked image modeling (MIM), narrow this domain gap by exploiting unlabeled real text images. This study first analyzes the original Masked AutoEncoder (MAE) and observes that random patch masking predominantly captures low-level textural features but misses high-level contextual representations. To fully exploit the high-level contextual representations, we introduce random blockwise and span masking in the text recognition task. These strategies can mask the continuous image patches and completely remove some characters, forcing the model to infer relationships among characters within a word. Our Multi-Masking Strategy (MMS) integrates random patch, blockwise, and span masking into the MIM frame, which jointly learns low and high-level textual representations. After fine-tuning with real data, MMS outperforms the state-of-the-art self-supervised methods in various text-related tasks, including text recognition, segmentation, and text-image super-resolution.

Figures (10)

• Figure 1: Illustration of (a) existing mask image modeling methods and (b) our proposed MMS that can use multiple masking strategies.
• Figure 2: A framework of Multi-Masking Strategy (MMS). Encoder and decoder parameters are shared between branches. During pre-training, a subset of image patches is masked (removed) by random patch masking, block masking, and span masking, respectively. The encoder only processes a subset of the visible patches. The decoder reconstructs images from the encoder output and mask tokens.
• Figure 3: Examples of images masked with different strategies. Different masking strategies force the model to learn different representations in mask image modeling.
• Figure 4: Reconstructions of scene text benchmarks images. From left to right: original image, masked image (top: random masking (75%); middle: block masking (50%); bottom: span masking (50%)), images reconstructed by MAE (random75%), images reconstructed by MAE (block 50%), images reconstructed by MAE (span 50%), images reconstructed by MMS.
• Figure 5: Visualization results of the attention map of the [CLS] token.