Instruction-Guided Scene Text Recognition

TL;DR

This work tackles scene text recognition by reframing it as instruction learning, where a lightweight instruction encoder and a cross-modal fusion module learn from rich $\langle condition,question,answer\rangle$ triplets that describe character attributes. The method supports multiple recognition pipelines (PR, AR, RI, ER) and can leverage attribute-prediction instructions for zeroshot-like inference, enabling robust performance with a small model (24.1M) and fast inference. Empirical results on English and Chinese benchmarks show state-of-the-art or competitive accuracy, with clear gains from instruction diversity and targeted strengthening for rare and morphologically similar characters. The approach offers practical benefits for edge/embedded deployment and points to future work on long-text recognition and OCR foundation-model integration, underscoring the value of structured, instruction-based cross-modal learning for fine-grained text understanding.

Abstract

Multi-modal models have shown appealing performance in visual recognition tasks, as free-form text-guided training evokes the ability to understand fine-grained visual content. However, current models cannot be trivially applied to scene text recognition (STR) due to the compositional difference between natural and text images. We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc. IGTR first devises $\left \langle condition,question,answer\right \rangle$ instruction triplets, providing rich and diverse descriptions of character attributes. To effectively learn these attributes through question-answering, IGTR develops a lightweight instruction encoder, a cross-modal feature fusion module and a multi-task answer head, which guides nuanced text image understanding. Furthermore, IGTR realizes different recognition pipelines simply by using different instructions, enabling a character-understanding-based text reasoning paradigm that differs from current methods considerably. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins, while maintaining a small model size and fast inference speed. Moreover, by adjusting the sampling of instructions, IGTR offers an elegant way to tackle the recognition of rarely appearing and morphologically similar characters, which were previous challenges. Code: https://github.com/Topdu/OpenOCR.

Figures (7)

• Figure 1: Upper: popular attention-based STR models follow the pipeline of positioning visual features for every character and then classifying. Mis-recognitions may happen if features are positioned incorrectly. Bottom: IGTR comprehends the text image first and then recognition. It understands the question and associates its answer with the corresponding visual features, generating robust instruction-guided STR.
• Figure 2: Overview of Instruction-Guided STR. Instruction triplets $\left \langle condition,question,answer\right \rangle$ are sampled from text ARTETA, where condition and question are encoded as the corresponding embeddings by the instruction encoder. Meanwhile, the image embedding is extracted from the image encoder. The three embeddings are interacted and fused by the cross-modal feature fusion module, and a multi-task answer head is appended to answer different types of questions. The whole architecture enjoys a lightweight design and ent-to-end optimization. Best viewed in color.
• Figure 3: Details of the instruction encoder. It maps different instructions to feature embeddings by using the five learnable embedding layers placed in the middle of the figure, where different attribute elements or additional tokens are summed to generate a unique condition or question representation. All condition embeddings are concatenated to generate a partition-level representation, and all question embeddings are grouped into four classes according to their answer types. Best viewed in color.
• Figure 4: The illustration of different recognition procedures by using different recognition instructions. [E] indicates the end symbol.
• Figure 5: Two illustrations of using different attribute prediction instructions described in Sec. 4.3 for text inference of image MOMS. The $condition$ is set to None.