Sequential Visual and Semantic Consistency for Semi-supervised Text Recognition

TL;DR

This work targets the data-hungry nature of scene text recognition (STR) by proposing a semi-supervised framework that leverages unlabeled real images alongside synthetic labels. It introduces word-level visual consistency via a shortest-path dynamic programming alignment of sequential visual features and word-level semantic consistency through reinforcement learning with FastText-based embeddings, built on a Mean-Teacher CCR baseline. The approach yields strong improvements over prior SSL methods, especially on challenging benchmarks, by mitigating sequence misalignment and enabling semantic supervision. Practically, it reduces the need for labeled data while enhancing robustness to distortions, occlusions, and domain gaps, and points toward tighter integration with language models for even better performance.

Abstract

Scene text recognition (STR) is a challenging task that requires large-scale annotated data for training. However, collecting and labeling real text images is expensive and time-consuming, which limits the availability of real data. Therefore, most existing STR methods resort to synthetic data, which may introduce domain discrepancy and degrade the performance of STR models. To alleviate this problem, recent semi-supervised STR methods exploit unlabeled real data by enforcing character-level consistency regularization between weakly and strongly augmented views of the same image. However, these methods neglect word-level consistency, which is crucial for sequence recognition tasks. This paper proposes a novel semi-supervised learning method for STR that incorporates word-level consistency regularization from both visual and semantic aspects. Specifically, we devise a shortest path alignment module to align the sequential visual features of different views and minimize their distance. Moreover, we adopt a reinforcement learning framework to optimize the semantic similarity of the predicted strings in the embedding space. We conduct extensive experiments on several standard and challenging STR benchmarks and demonstrate the superiority of our proposed method over existing semi-supervised STR methods.

Figures (3)

• Figure 1: Overall pipeline of our proposed method. A regular recognizer is trained in a supervised manner with labeled synthetic images. A Mean-Teacher architecture is employed for unlabeled real data to conduct unsupervised learning via consistency regularization (CR). Besides the CCR on the character-level probability distribution, word-level visual CR on the glimpse vectors and semantic CR on the predicted strings are introduced. Best viewed in color.
• Figure 2: The comparison between the word-level consistencies calculated via one-to-one character matching and dynamic programming. The characters displayed at the left and top of the grid represent the predicted strings from Teacher Recognizer and Student Recognizer, respectively. In the grid, the red block indicates an incorrect matching while an azure block represents a correct matching.
• Figure 3: Visualization of text recognition results. The four strings near each image represent the prediction of Baseline, Baseline+CCR, Baseline+CCR+WVCR, and Baseline+CCR+WVCR+WSCR respectively. (a), (b), (c) are different image groups to demonstrate the effectiveness of each model unit.