TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision

TL;DR

This work tackles end-to-end scene text spotting by introducing TextFormer, a Transformer-based, query-driven framework that unifies detection, segmentation, and recognition without RoI operations. It advances the field with an Adaptive Global aGgregation (AGG) module for reading arbitrarily-shaped text and a mixed-supervision strategy that leverages weak annotations to boost performance while reducing labeling costs. TextFormer achieves state-of-the-art results on English and Chinese benchmarks, notably delivering a $1 ext{-}NED$ improvement of $13.2\%$ on the challenging TDA-ReCTS dataset, driven by deep multi-task feature sharing. The combination of multi-task learning, global feature aggregation, and mixed supervision has practical significance for robust, multilingual text understanding in real-world applications.

Abstract

End-to-end text spotting is a vital computer vision task that aims to integrate scene text detection and recognition into a unified framework. Typical methods heavily rely on Region-of-Interest (RoI) operations to extract local features and complex post-processing steps to produce final predictions. To address these limitations, we propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. Specifically, using query embedding per text instance, TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing without sacrificing flexibility or simplicity. Additionally, we design an Adaptive Global aGgregation (AGG) module to transfer global features into sequential features for reading arbitrarily-shaped texts, which overcomes the sub-optimization problem of RoI operations. Furthermore, potential corpus information is utilized from weak annotations to full labels through mixed supervision, further improving text detection and end-to-end text spotting results. Extensive experiments on various bilingual (i.e., English and Chinese) benchmarks demonstrate the superiority of our method. Especially on TDA-ReCTS dataset, TextFormer surpasses the state-of-the-art method in terms of 1-NED by 13.2%.

Figures (8)

• Figure 1: Comparison of previous end-to-end text spotting methods with our query-based pipeline. (a) shows the traditional top-down (two-stage) framework, which uses RoI operation to extract features and integrate the detection and recognition branches. (b) depicts the bottom-up framework (one-stage), which predicts all possible characters and requires intricate post-processing step to match all characters into the correct words. (c) is our query-based framework that directly outputs both segmentation and recognition results without RoI operation and post-processing step.
• Figure 2: Examples of images with different annotations. (a) is the full annotation that includes text regions with text transcription labels. (b) shows the text annotation that contains text transcription annotations without location information. (c) indicates the weak annotation that only the transcription of text-of-interest is provided. The regions commonly referred to as "Text-of-interest" typically contain the names of store fonts or descriptions of landmarks, supplying significant information for localization and navigation purposes sun2019icdar.
• Figure 3: The overall architecture of TextFormer. We propose a query-based end-to-end text spotter where each query represents a latent text instance. Our mixed-supervision training scheme allows our network to train with a mix of fully and weakly labeled data for better results. Note the composition of the loss function for training weakly supervised data is framed by the dashed line.
• Figure 4: The detailed structure of our recognition branch. It is composed of a feature extractor and an attention-based recognizer. The AGG module extracts the sequential features from the shared semantic features. The Transformer-based recognition head predicts characters for the corresponding character queries.
• Figure 5: Visualization examples for ICDAR 2015 dataset without lexicons. Best zoom in for better visualization.