Exploring OCR-augmented Generation for Bilingual VQA

TL;DR

The paper investigates OCR-augmented generation for multilingual Visual Question Answering by integrating OCR capabilities into Vision Language Models. It introduces KLOCR, an open-source bilingual OCR model trained on a 100M dataset, and KOCRBench, a Korean VQA benchmark, and demonstrates that OCR-extracted text markedly boosts VLM performance across both open-source and commercial models. A key finding is that character-accurate information from OCR is crucial for accurate answers, especially for Key Information Extraction tasks. The work provides datasets, models, and empirical insights to advance multilingual VQA and identifies directions for improving OCR accuracy and reasoning in VLMs.

Abstract

We investigate OCR-augmented generation with Vision Language Models (VLMs), exploring tasks in Korean and English toward multilingualism. To support research in this domain, we train and release KLOCR, a strong bilingual OCR baseline trained on 100M instances to augment VLMs with OCR ability. To complement existing VQA benchmarks, we curate KOCRBench for Korean VQA, and analyze different prompting methods. Extensive experiments show that OCR-extracted text significantly boosts performance across open source and commercial models. Our work offers new insights into OCR-augmented generation for bilingual VQA. Model, code, and data are available at https://github.com/JHLee0513/KLOCR.

Figures (5)

• Figure 1: OCR model comparison on the validation set of KLOCR data. KLOCR not only sets state-of-the-art accuracy on the benchmark, but also exhibits the best accuracy-speed tradeoff.
• Figure 2: Samples from KLOCR data mixture. The data collection is bilingual and varies across multiple domains (e.g. documents, road signs, handwriting).
• Figure 3: Sample images from the KOCRBench dataset. We collect various samples from KLOCR data mixture and repurpose samples from KVQA to create (image, question, answer) triplets. The dataset covers various scenarios with road signs, product images, and documents. Images have been resized for visualization purposes.
• Figure 4: Example failure case of miscounting. Blue text indicates translated text for context. Boxed areas with red text highlight three applications written down. When asked to count the number of applicants in the form, VLMs often response to mistakenly list 5 valid applicants instead of 3.
• Figure 5: KLOCR data processing.