Inverse-like Antagonistic Scene Text Spotting via Reading-Order Estimation and Dynamic Sampling

TL;DR

This work addresses inverse-like scene text spotting by introducing IATS, a unified end-to-end framework that jointly models reading-order and adaptive feature sampling. A reading-order estimation module (REM) extracts ordering information from the initial boundary produced by an initial boundary module (IBM) and is trained via a joint loss that includes classification, orthogonality, and distribution terms. A dynamic sampling module (DSM) uses a thin-plate spline to flexibly sample recognition features, mitigating dependence on detection accuracy. The boundary refinement module (BRM) iteratively refines boundaries, while the entire system integrates seamlessly with the recognition module for end-to-end training. Extensive experiments across regular and inverse-like datasets demonstrate significant improvements, especially on datasets with complex layouts, validating the importance of reading-order information and dynamic sampling in text spotting.

Abstract

Scene text spotting is a challenging task, especially for inverse-like scene text, which has complex layouts, e.g., mirrored, symmetrical, or retro-flexed. In this paper, we propose a unified end-to-end trainable inverse-like antagonistic text spotting framework dubbed IATS, which can effectively spot inverse-like scene texts without sacrificing general ones. Specifically, we propose an innovative reading-order estimation module (REM) that extracts reading-order information from the initial text boundary generated by an initial boundary module (IBM). To optimize and train REM, we propose a joint reading-order estimation loss consisting of a classification loss, an orthogonality loss, and a distribution loss. With the help of IBM, we can divide the initial text boundary into two symmetric control points and iteratively refine the new text boundary using a lightweight boundary refinement module (BRM) for adapting to various shapes and scales. To alleviate the incompatibility between text detection and recognition, we propose a dynamic sampling module (DSM) with a thin-plate spline that can dynamically sample appropriate features for recognition in the detected text region. Without extra supervision, the DSM can proactively learn to sample appropriate features for text recognition through the gradient returned by the recognition module. Extensive experiments on both challenging scene text and inverse-like scene text datasets demonstrate that our method achieves superior performance both on irregular and inverse-like text spotting.

Figures (16)

• Figure 1: Illustrations of different text feature sampling methods: (a) Masked RoI: a shape mask is used to formulate text regions, while background noise can be suppressed; (b) TPS without reading-order: the text is transformed into a horizontal region using boundary control points to generate fixed sample grids; (c) TPS with reading-order; (d) DSM with reading-order.
• Figure 2: Comparison of TPS and DSM.(a) Boundary control points with reading-order; (b) TPS: generating a fixed and regularly sampling grids heavily relay on control points; (c) DSM: dynamically generating adaptive sampling grids through self-adjustment with recognition model; (d) Visual comparison of sample grids for TPS and DSM.
• Figure 3: Overview of the proposed framework. The orange lines indicate the detection flow, and the blue lines indicate the recognition flow. The predictions of the different modules are also visualized in the origin image.
• Figure 4: (a) Architecture of initial boundary module; (b) The generation of initial text boundaries.
• Figure 5: (a) The original label form of text boundary implies the reading-order. (b) The original label form induces the detector to implicitly learn the reading order, resulting in false positives and jagged edges. (c) Even with extensive rotation augmentation during training, the detector still can't learn the reading-order well. (d) Text Perceptron TextPerception uses order-aware segmentation to indicate the head and tail of text instances and capture latent reading-orders. (e) PGNet PGNet uses text direction offset (TDO) maps to extract the text reading-order. (f) Our method uses the four key corners on the text boundary to indicate the reading-order.