Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer

TL;DR

E2 STR demonstrates that a regular-sized model is sufficient to achieve effective ICL capabilities in STR, and exhibits remarkable training-free adaptation in var-ious scenarios and outperforms even the fine-tuned state-of-the-art approaches on public benchmarks.

Abstract

Scene text recognition (STR) in the wild frequently encounters challenges when coping with domain variations, font diversity, shape deformations, etc. A straightforward solution is performing model fine-tuning tailored to a specific scenario, but it is computationally intensive and requires multiple model copies for various scenarios. Recent studies indicate that large language models (LLMs) can learn from a few demonstration examples in a training-free manner, termed "In-Context Learning" (ICL). Nevertheless, applying LLMs as a text recognizer is unacceptably resource-consuming. Moreover, our pilot experiments on LLMs show that ICL fails in STR, mainly attributed to the insufficient incorporation of contextual information from diverse samples in the training stage. To this end, we introduce E$^2$STR, a STR model trained with context-rich scene text sequences, where the sequences are generated via our proposed in-context training strategy. E$^2$STR demonstrates that a regular-sized model is sufficient to achieve effective ICL capabilities in STR. Extensive experiments show that E$^2$STR exhibits remarkable training-free adaptation in various scenarios and outperforms even the fine-tuned state-of-the-art approaches on public benchmarks. The code is released at https://github.com/bytedance/E2STR .

Figures (11)

• Figure 1: Demonstration of real-world scene text scenarios and the adaptation pipeline. (a) Diversified scenarios of scene text in the real world. (b) The adaptation pipeline of current methods. They typically have to fine-tune upon a trained STR model with the training set, under a specific scenario. (c) The adaptation pipeline of our proposed E$^2$STR. Our method automatically selects in-context prompts and performs training-free adaptation when faced with novel scenarios. Blue characters denote ambiguous scene text that is easily misrecognized.
• Figure 2: Our pilot experiments. (a) The randomly concatenated scene text sequence. (b) Our proposed context-rich scene text sequence. (c) By training an STR model based on the randomly concatenated scene text sequence, we evaluate the model on three cross-domain datasets.
• Figure 3: Pipeline of our E$^2$STR. Top: E$^2$STR is trained with our in-context training strategy to obtain the ICL capability. Down: During inference, E$^2$STR selects in-context prompts based on a kNN strategy, then the test sample grasps context information from the prompts to assist the recognition. Specifically, the ambiguous character "a" in the test sample is easily misrecognized as "q". With the vision-language context produced by the in-context prompts ( i.e., "a" in the first in-context prompt), E$^2$STR rectifies the result. Note that in practice the in-context pool maintains image tokens and thus does not need to go through the vision encoder.
• Figure 4: Illustration of the split strategy, the transform strategy, and how we hybrid them in practice.
• Figure 5: Comparison with the fine-tuned models. We report the average performance on three cross-domain datasets. Please note that ABINet abinet, SATRN satrn and MAERec Union14M are fine-tuned with the in-domain data, while our E$^2$STR-ICL is training-free.