An Effective Data Augmentation Method by Asking Questions about Scene Text Images

TL;DR

A VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks that aligns visual features with textual queries to jointly reason over images and questions.

Abstract

Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at https://github.com/xuyaooo/DataAugOCR.

Figures (1)

• Figure 1: Overall architecture of the proposed VQA-based OCR framework showing the integration of visual features from TrOCR encoder with textual query embeddings through cross-modal attention.