SemiETS: Integrating Spatial and Content Consistencies for Semi-Supervised End-to-end Text Spotting

TL;DR

SemiETS tackles semi-supervised end-to-end text spotting by addressing two central issues: inconsistent pseudo labels across detection and recognition, and misalignment between teacher and student signals. It introduces Progressive Sample Assignment to produce reliable hierarchical supervision and Mutual Mining Strategy to exploit the reciprocal information between detection and recognition via spatial-aware and content-aware cues. The approach yields state-of-the-art results across Total-Text, ICDAR 2015, and CTW1500, especially under very low labeled-data regimes, and remains effective with fully labeled data and across different spotters and domain shifts. The framework’s cross-task synergy and adaptive weighting provide practical improvements for real-world scene text spotting with limited annotations, and its design invites future multilingual and more diverse script extensions.

Abstract

Most previous scene text spotting methods rely on high-quality manual annotations to achieve promising performance. To reduce their expensive costs, we study semi-supervised text spotting (SSTS) to exploit useful information from unlabeled images. However, directly applying existing semi-supervised methods of general scenes to SSTS will face new challenges: 1) inconsistent pseudo labels between detection and recognition tasks, and 2) sub-optimal supervisions caused by inconsistency between teacher/student. Thus, we propose a new Semi-supervised framework for End-to-end Text Spotting, namely SemiETS that leverages the complementarity of text detection and recognition. Specifically, it gradually generates reliable hierarchical pseudo labels for each task, thereby reducing noisy labels. Meanwhile, it extracts important information in locations and transcriptions from bidirectional flows to improve consistency. Extensive experiments on three datasets under various settings demonstrate the effectiveness of SemiETS on arbitrary-shaped text. For example, it outperforms previous state-of-the-art SSL methods by a large margin on end-to-end spotting (+8.7%, +5.6%, and +2.6% H-mean under 0.5%, 1%, and 2% labeled data settings on Total-Text, respectively). More importantly, it still improves upon a strongly supervised text spotter trained with plenty of labeled data by 2.0%. Compelling domain adaptation ability shows practical potential. Moreover, our method demonstrates consistent improvement on different text spotters.

Figures (10)

• Figure 1: Comparison of different semi-supervised tasks related to text, including (a) semi-supervised text detection, (b) semi-supervised text recognition, and (c) semi-supervised text spotting. The red arrow indicates the supervision flow of pseudo labels.
• Figure 2: Illustration of the inconsistency issues including inconsistency (a) between tasks, and (b) between teacher and student.
• Figure 3: Overview of the proposed framework where the labeled data flow is omitted. Given an unlabeled image, Progressive Sample Assignment selects reliable pseudo labels and splits them into Det-only and E2E labels. Then, the Mutual Mining Strategy explores effective information in E2E labels in a crossover strategy with Spatial-aware Consistency Integration and Content-aware Region Calibration.
• Figure 4: The details of SCI and CRC in the proposed Mutual Mining Strategy. We distinguish text instances using green and blue in (b) and the wrongly recognized characters indicated in red.
• Figure 5: Statistical analysis on the relationship of the accuracy of detected regions and text similarity.