InstructOCR: Instruction Boosting Scene Text Spotting

TL;DR

InstructOCR introduces an instruction-based framework for scene text spotting that aligns human language instructions with visual text. It employs a ResNet-50 image encoder and a BERT-based text encoder within an encoder-decoder architecture, generating per-text-instance sequences represented as $[x,y,t]$, with cross-attention fusing instruction cues. A set of ten templates derived from text attributes provides diverse supervisory signals, enabling strong OCR performance and transfer to scene-text VQA, achieving $+2.6\%$ on TextVQA and $+2.1\%$ on ST-VQA after pretraining with instructions. The work demonstrates SOTA results on multiple benchmarks and highlights the value of language-grounded instruction signals for OCR, while noting limitations related to training data scale for VQA tasks.

Abstract

In the field of scene text spotting, previous OCR methods primarily relied on image encoders and pre-trained text information, but they often overlooked the advantages of incorporating human language instructions. To address this gap, we propose InstructOCR, an innovative instruction-based scene text spotting model that leverages human language instructions to enhance the understanding of text within images. Our framework employs both text and image encoders during training and inference, along with instructions meticulously designed based on text attributes. This approach enables the model to interpret text more accurately and flexibly. Extensive experiments demonstrate the effectiveness of our model and we achieve state-of-the-art results on widely used benchmarks. Furthermore, the proposed framework can be seamlessly applied to scene text VQA tasks. By leveraging instruction strategies during pre-training, the performance on downstream VQA tasks can be significantly improved, with a 2.6% increase on the TextVQA dataset and a 2.1% increase on the ST-VQA dataset. These experimental results provide insights into the benefits of incorporating human language instructions for OCR-related tasks.

Figures (5)

• Figure 1: Examples of text recognition results generated using various instructions. This illustrates how different instructions can influence the output of the text recognition process.
• Figure 2: Main framework of InstructOCR. InstructOCR is an encoder-decoder architecture, with input branches consisting of an image encoder and a text encoder that handle visual and textual features separately.
• Figure 3: Examples of decoder output for scene text spotting and VQA tasks.
• Figure 4: Visual results on Total-Text, ICDAR2015 and ICDAR2013. Our model can effectively handle curved, distorted, and blurred scene text.
• Figure 5: Visualization of correct recognition results on the scene text spotting and VQA datasets after incorporating instructions. The first row corresponds to the scene text spotting dataset, while the second row corresponds to the VQA dataset. "W/" and "W/O" indicate whether OCR training was conducted with or without instructions, respectively.