ViTextVQA: A Large-Scale Visual Question Answering Dataset for Evaluating Vietnamese Text Comprehension in Images

TL;DR

This work introduces ViTextVQA, a large-scale Vietnamese dataset focused on scene-text understanding for visual question answering, featuring 16,762 images and 50,342 QA pairs. It proposes ViTextBLIP-2, a multimodal fusion framework that freezes vision and language backbones (ViT, SwinTextSpotter OCR, and ViT5 LM) while training a Q-Former to fuse multimodal signals, achieving state-of-the-art results (F1 = 53.95, EM = 25.48). A key finding is that sorting OCR tokens top-left to bottom-right improves performance, likely aligning with Vietnamese reading order, while OCR token arrangement and text size significantly influence results. The paper also provides a thorough analysis of diacritics and Vietnamese word segmentation and demonstrates that current LLMs lag behind human performance on Vietnamese text-based VQA, suggesting avenues for VQG and prompting-based approaches to advance this niche yet important language domain.

Abstract

Visual Question Answerinng (VQA) is a complicated task that requires the capability of simultaneously processing natural language and images. This task was initially researched with a focus on developing methods to help machines understand objects and scene contexts in images. However, some scene text that carries explicit information about the full content of the image is not mentioned. Along with the continuous development of the AI era, there have been many studies on the reading comprehension ability of VQA models in the world. Therefore, we introduce the first large-scale dataset in Vietnamese specializing in the ability to understand scene text, we call it ViTextVQA (\textbf{Vi}etnamese \textbf{Text}-based \textbf{V}isual \textbf{Q}uestion \textbf{A}nswering dataset) which contains \textbf{over 16,000} images and \textbf{over 50,000} questions with answers. To tackle this task efficiently, we propose ViTextBLIP-2, an novel multimodal feature fusion Method, which optimizes Vietnamese OCR-based VQA by integrating a frozen Vision Transformer, SwinTextSpotter OCR, and ViT5 LLM with a trainable Q-Former for multimodal feature fusion. Through experiments with various state-of-the-art models, we uncover the significance of the order in which tokens in OCR text are processed and selected to formulate answers. This finding helped us significantly improve the performance of the baseline models on the ViTextVQA dataset. Our dataset is available (https://github.com/minhquan6203/ViTextVQA-Dataset) for research purposes.

Figures (22)

• Figure 1: Samples extracted from the ViTextVQA dataset.
• Figure 2: The overview process of creating our dataset ViTextVQA.
• Figure 3: User interface of VQALabeling.
• Figure 4: *.json file storage structure.
• Figure 5: Statistic of images and question-answer pairs.