SwinTextSpotter v2: Towards Better Synergy for Scene Text Spotting

TL;DR

SwinTextSpotter v2 tackles the core challenge of end-to-end scene text spotting by strengthening the interaction between detection and recognition through Recognition Conversion (RC) and Recognition Alignment (RA), and by simplifying the detector with a Box Selection Schedule. The method uses a query-based detector and a Swin-Transformer backbone, enabling dynamic feature alignment and targeted feature injection into recognition without requiring character-level annotations or rectification modules. It achieves state-of-the-art results across multilingual and arbitrarily-shaped text benchmarks, while significantly reducing detector parameters and maintaining or improving end-to-end accuracy. The work demonstrates robust scalability, including improvements when leveraging larger training data such as TextOCR, and provides a practical, efficient framework for real-world scene text understanding.

Abstract

End-to-end scene text spotting, which aims to read the text in natural images, has garnered significant attention in recent years. However, recent state-of-the-art methods usually incorporate detection and recognition simply by sharing the backbone, which does not directly take advantage of the feature interaction between the two tasks. In this paper, we propose a new end-to-end scene text spotting framework termed SwinTextSpotter v2, which seeks to find a better synergy between text detection and recognition. Specifically, we enhance the relationship between two tasks using novel Recognition Conversion and Recognition Alignment modules. Recognition Conversion explicitly guides text localization through recognition loss, while Recognition Alignment dynamically extracts text features for recognition through the detection predictions. This simple yet effective design results in a concise framework that requires neither an additional rectification module nor character-level annotations for the arbitrarily-shaped text. Furthermore, the parameters of the detector are greatly reduced without performance degradation by introducing a Box Selection Schedule. Qualitative and quantitative experiments demonstrate that SwinTextSpotter v2 achieved state-of-the-art performance on various multilingual (English, Chinese, and Vietnamese) benchmarks. The code will be available at \href{https://github.com/mxin262/SwinTextSpotterv2}{SwinTextSpotter v2}.

Figures (8)

• Figure 1: The effectiveness of Recognition Conversion (RC) and Recognition Alignment (RA). Previous connector-based methods usually struggle in the misalignment issue. Too few sampling points result in the failure to capture certain characters in the text region, leading to a loss of critical information. Too many sampling points produce a lot of background noise, thereby misleading the recognizer. Recognition Alignment can dynamically sample the text feature.
• Figure 2: Overview of several end-to-end scene text spotting methods that correspond to our research objectives. Within the ground-truth (GT) box, the letters 'W' and 'C' indicate word-level and character-level annotations, respectively. The letters 'H', 'Q', and 'A' signify the method's ability to detect horizontal, quadrilateral, and arbitrarily-shaped text, respectively. Conversely, the dashed box denotes the shape of the text that the method cannot detect. Figure style from liu2020abcnetwang2021pgnetzhang2022text
• Figure 3: The framework of the proposed SwinTextSpotter v2. The gray arrows denote the feature extraction from images. The green arrows and blue arrows represent the detection stage and the recognition stage, respectively. The outputs of the detection head are refined in K stages. The output detection in the $K^{th}$ stage serves as the input to the recognition stage.
• Figure 4: Illustration of $k^{th}$ stage in detection. $f_{k-1}^{prop}$ is the proposal features output by previous stage. $MSA_{k}$ refers to the multi-head-attention in $k^{th}$ stage. $f_{k}^{prop}$ will serve as the input to next stage.
• Figure 5: The architecture of Recognition Alignment and visualizations of global- and local-level alignment attention maps. The red star indicates the query point.