Efficient and Accurate Scene Text Recognition with Cascaded-Transformers

TL;DR

The paper tackles the high computational burden of scene text recognition by introducing cascaded-transformers that progressively prune vision tokens in the encoder while preserving a rich latent representation for decoding. The encoder is decomposed into multiple sub-models, each reducing the token count and selecting the most informative tokens, with four reduction strategies analyzed. The decoder remains a Permuted-Language Decoder, and the system is trained with a next-permuted-token objective. Empirical results across various cascaded configurations show substantial reductions in GFLOPs (often by factors of 2–10) with only small or negligible drops in word accuracy on standard STR benchmarks, and strong robustness on real-world distorted text, outperforming a SOTA baseline (CLIP4STR-B) and approaching larger variants (CLIP4STR-L) at a fraction of the computational cost. This approach offers a practical path to deploy accurate STR on resource-constrained devices while maintaining competitive performance.

Abstract

In recent years, vision transformers with text decoder have demonstrated remarkable performance on Scene Text Recognition (STR) due to their ability to capture long-range dependencies and contextual relationships with high learning capacity. However, the computational and memory demands of these models are significant, limiting their deployment in resource-constrained applications. To address this challenge, we propose an efficient and accurate STR system. Specifically, we focus on improving the efficiency of encoder models by introducing a cascaded-transformers structure. This structure progressively reduces the vision token size during the encoding step, effectively eliminating redundant tokens and reducing computational cost. Our experimental results confirm that our STR system achieves comparable performance to state-of-the-art baselines while substantially decreasing computational requirements. In particular, for large-models, the accuracy remains same, 92.77 to 92.68, while computational complexity is almost halved with our structure.

Figures (4)

• Figure 1: The overall architecture of our STR system. Our system consists of encoder $e$ and decoder $d$ models and predicts text characters $\mathbf{y}$ from an input image $\mathbf{x}$.
• Figure 2: The flows of standard vision transformer and proposed cascaded transformers. Both models take an input image $\mathbf{x}$ and output vision tokens $\mathbf{v}$. The key component of our proposed model is a series of cascaded transformers, which reduces the number of vision tokens at each level by selectively retaining a subset of the most relevant ones for scene text recognition (i.e., $N < N_C$). Details are explain in Section \ref{['subsection:ect']}.
• Figure 3: Four different reduction styles for cascaded transformers: (a) early-level reduction (b) mid-level reduction (c) late-level reduction, and (d) multi-level reduction.
• Figure 4: Prediction results of our STR system on real-world examples by visualizing character attention maps separately.