Towards Unified Multi-granularity Text Detection with Interactive Attention

TL;DR

This work presents Detect Any Text (DAT), a unifiedTransformer-based framework that jointly handles word, line, paragraph, and page text detection, overcoming the need for separate models. It introduces an interactive across-granularity attention mechanism that propagates information across granularities and uses mixed-granularity training to leverage datasets with incomplete annotations. A prompt-based segmentation module further refines localization of arbitrarily-shaped text and document pages. Empirically, DAT achieves state-of-the-art results across scene text detection, document layout analysis, and page segmentation benchmarks while maintaining competitive efficiency, highlighting its practical value for end-to-end text understanding in diverse real-world scenarios.

Abstract

Existing OCR engines or document image analysis systems typically rely on training separate models for text detection in varying scenarios and granularities, leading to significant computational complexity and resource demands. In this paper, we introduce "Detect Any Text" (DAT), an advanced paradigm that seamlessly unifies scene text detection, layout analysis, and document page detection into a cohesive, end-to-end model. This design enables DAT to efficiently manage text instances at different granularities, including *word*, *line*, *paragraph* and *page*. A pivotal innovation in DAT is the across-granularity interactive attention module, which significantly enhances the representation learning of text instances at varying granularities by correlating structural information across different text queries. As a result, it enables the model to achieve mutually beneficial detection performances across multiple text granularities. Additionally, a prompt-based segmentation module refines detection outcomes for texts of arbitrary curvature and complex layouts, thereby improving DAT's accuracy and expanding its real-world applicability. Experimental results demonstrate that DAT achieves state-of-the-art performances across a variety of text-related benchmarks, including multi-oriented/arbitrarily-shaped scene text detection, document layout analysis and page detection tasks.

Figures (9)

• Figure 1: Illustration of the structural correlations among multi-granularity text instances, i.e., word(annotated with yellow polygons), text-line(annotated with green polygons), paragraph(annotated with brown polygons) and page(annotated with magenta contours). The blurred small text instances are ignored.
• Figure 2: Network structure of Detect Any Text (DAT). "DET" illustrates the multi-granularity detection framework with a single layer of Transformer decoder network, where the residual connection and norm layers are omitted for simplicity. "SEG" illustrates the model pipeline of prompt-based segmentation module.
• Figure 3: Illustration of across-granularity representation learning with interactive attention module (interaction factor $\mathcal{I}=1$).
• Figure 4: Visualization results of DAT on each granularity of benchmark datasets. "DET" and "SEG" indicate text detection and segmentation results respectively. For multi-oriented datasets ICDAR-2015 and MSRA-TD500, the DAT-SEG model further refined detection results, particularly for curved texts. However, a slight decline in benchmark evaluation results occurred due to the quadrilateral-based annotations.
• Figure 5: Multi-granularity pseudo labels produced by DAT. From left to right: text detection results at the word, line and paragraph levels. Note that these datasets do not have the corresponding GT annotations for these specific granularities.