LEGO: Self-Supervised Representation Learning for Scene Text Images

TL;DR

The paper tackles data scarcity and domain shift in scene text recognition by introducing LEGO, a self-supervised framework that leverages unlabeled real data through three text-tailored pretext tasks—Selective Individual Discrimination, Modified Masked Image Modeling, and Random-ordered Text Rearrangement—guided by a Text Knowledge Codebook derived from a frozen T-VQVAE. The codebook provides consistent text priors to filter negatives, guide reconstruction, and ground truth ordering, enabling richer representations for downstream recognition and super-resolution. Empirical results show LEGO surpasses prior SSL methods on standard benchmarks in both probe and semi-supervised evaluations and delivers strong gains in scene text super-resolution, highlighting its versatility and practical impact for real-world text understanding systems.

Abstract

In recent years, significant progress has been made in scene text recognition by data-driven methods. However, due to the scarcity of annotated real-world data, the training of these methods predominantly relies on synthetic data. The distribution gap between synthetic and real data constrains the further performance improvement of these methods in real-world applications. To tackle this problem, a highly promising approach is to utilize massive amounts of unlabeled real data for self-supervised training, which has been widely proven effective in many NLP and CV tasks. Nevertheless, generic self-supervised methods are unsuitable for scene text images due to their sequential nature. To address this issue, we propose a Local Explicit and Global Order-aware self-supervised representation learning method (LEGO) that accounts for the characteristics of scene text images. Inspired by the human cognitive process of learning words, which involves spelling, reading, and writing, we propose three novel pre-text tasks for LEGO to model sequential, semantic, and structural features, respectively. The entire pre-training process is optimized by using a consistent Text Knowledge Codebook. Extensive experiments validate that LEGO outperforms previous scene text self-supervised methods. The recognizer incorporated with our pre-trained model achieves superior or comparable performance compared to state-of-the-art scene text recognition methods on six benchmarks. Furthermore, we demonstrate that LEGO can achieve superior performance in other text-related tasks.

Figures (6)

• Figure 1: Problems encountered when performing self-supervised learning (SSL) on scene text images. (a) Ambiguity brought by rough sample division strategy of contrastive learning, (b) Indeterminacy due to invalid reconstruction targets caused by random masking, and (c) Sequentiality, a natural property of text that the same letters in different orders can form words with different meanings.
• Figure 2: The pipeline of LEGO. The T-Encoder depicted on the left is derived from T-VQVAE training. Before SSL pre-training, we feed input images into the T-Encoder to obtain the unified Text Knowledge Codebook. After three pretext tasks (shown on the right) designed for scene text SSL, the pre-trained ViT encoder can be migrated to downstream tasks.
• Figure 3: The architecture of T-VQVAE. The quantizer aggregates de-styled text features in the latent vector space to generate the Text Knowledge Codebook through tokenization and retrieval. An officially trained VGG-16 is employed to calculate the perceptual loss.
• Figure 4: Samples in the Text Knowledge Codebook. Three indexes correspond to parts of the letter 'O', 'A', and 'S' respectively.
• Figure 5: Three SSL pretext tasks for LEGO: (a) Selective Individual Discrimination task, (b) modified Masked Image Modeling task, and (c) Random-ordered Text Rearrangement task. These three tasks are all facilitated by the knowledge from our Text Knowledge Codebook.