Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark

TL;DR

This paper introduces ViSignVQA, the first large-scale Vietnamese signboard VQA dataset with 10,762 images and 25,573 QA pairs, capturing real-world bilingual signboard content. It benchmarks OCR-integrated baselines by incorporating SwinTextSpotter and ViT5, and demonstrates substantial gains when OCR text augments questions, with a GPT-4–powered multi-agent VQA system achieving 75.98% accuracy. The work provides an in-depth analysis of question/answer properties, signboard features, and regional variation, highlighting the challenges of Vietnamese signboards for multimodal reasoning. By offering a new resource and benchmarks, it paves the way for OCR-grounded VQA in Vietnamese and motivates domain-specific multimodal methods, with future directions including visual question generation and a Vietnamese multimodal chatbot built atop ViSignVQA.

Abstract

Understanding signboard text in natural scenes is essential for real-world applications of Visual Question Answering (VQA), yet remains underexplored, particularly in low-resource languages. We introduce ViSignVQA, the first large-scale Vietnamese dataset designed for signboard-oriented VQA, which comprises 10,762 images and 25,573 question-answer pairs. The dataset captures the diverse linguistic, cultural, and visual characteristics of Vietnamese signboards, including bilingual text, informal phrasing, and visual elements such as color and layout. To benchmark this task, we adapted state-of-the-art VQA models (e.g., BLIP-2, LaTr, PreSTU, and SaL) by integrating a Vietnamese OCR model (SwinTextSpotter) and a Vietnamese pretrained language model (ViT5). The experimental results highlight the significant role of the OCR-enhanced context, with F1-score improvements of up to 209% when the OCR text is appended to questions. Additionally, we propose a multi-agent VQA framework combining perception and reasoning agents with GPT-4, achieving 75.98% accuracy via majority voting. Our study presents the first large-scale multimodal dataset for Vietnamese signboard understanding. This underscores the importance of domain-specific resources in enhancing text-based VQA for low-resource languages. ViSignVQA serves as a benchmark capturing real-world scene text characteristics and supporting the development and evaluation of OCR-integrated VQA models in Vietnamese.

Figures (26)

• Figure 1: Examples Retrieved from the ViSignVQA Dataset.
• Figure 2: ViSignVQA Dataset Creation Overview.
• Figure 3: Distribution of Images and Question–Answer Pairs.
• Figure 4: Distribution of question length.
• Figure 5: Distribution of Postag in question.