Aggregated Text Transformer for Scene Text Detection

TL;DR

The paper addresses the challenge of detecting scene text with irregular shapes and varying scales by introducing ATTR, a multi-scale aggregated text Transformer. It leverages an image pyramid with shared-weight projections, a deformable-attention Transformer encoder, and scale-wise decoders to produce individual binary masks for text instances, enabling tight boundaries and separation of neighboring texts. Key contributions include a novel text-instance representation, effective multi-scale feature fusion within a Transformer framework, and strong empirical results on curved-text and multi-oriented benchmarks, alongside detailed ablations and speed analyses. The approach demonstrates that cross-scale self-attention on a pyramid can yield robust text representations and state-of-the-art or competitive performance across multiple public datasets.

Abstract

This paper explores the multi-scale aggregation strategy for scene text detection in natural images. We present the Aggregated Text TRansformer(ATTR), which is designed to represent texts in scene images with a multi-scale self-attention mechanism. Starting from the image pyramid with multiple resolutions, the features are first extracted at different scales with shared weight and then fed into an encoder-decoder architecture of Transformer. The multi-scale image representations are robust and contain rich information on text contents of various sizes. The text Transformer aggregates these features to learn the interaction across different scales and improve text representation. The proposed method detects scene texts by representing each text instance as an individual binary mask, which is tolerant of curve texts and regions with dense instances. Extensive experiments on public scene text detection datasets demonstrate the effectiveness of the proposed framework.

Figures (10)

• Figure 1: Scene text detection results with single-scale input (top) and with multi-scale aggregation (bottom). The single-scale setting may assign low confidence for the scene texts with small size.
• Figure 2: Overview of multi-scale aggregation strategies. (a) Initial results are computed separately from each image scale and the detection result is fused with a strategy like NMS. (b) Feature maps are obtained from a multi-scale image pyramid and fed into one detection head. (c) Aggregation of multi-scale feature maps for the detection results.
• Figure 3: The proposed framework. Given a scene image, we augment it into a multi-scale image pyramid. The multi-scale image representations are computed with a projection module with shared weights for different scales and a Transformer encoder that learns multi-scale contexts. The text instances are then detected with the decoder module.
• Figure 4: Taking feature embeddings from image as input, we first employ the Transformer Encoder to compute the multi-scale image representations. The text embedding is updated with the image representations as input.
• Figure 5: Text instance representation generation and the visualization of binary masks. Each result image indicates one mask (text instance) in the original image.