Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images

Huy Quang Pham, Thang Kien-Bao Nguyen, Quan Van Nguyen, Dan Quang Tran, Nghia Hieu Nguyen, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

TL;DR

ViOCRVQA addresses the underexplored OCR-VQA problem for Vietnamese by introducing the ViOCRVQA dataset (28,282 images with 123,781 QA pairs) and a multimodal VisionReader model. VisionReader fuses textual OCR tokens, object features from VinVL, and grid features from ViT within a transformer encoder-decoder to generate exact-text answers from book-cover images. The paper demonstrates that OCR quality and object context substantially influence performance, with VisionReader (ViT5 backbone) achieving the strongest EM of 41.16% and F1 of 69.90% on the test set, and significant gains in book-genre prediction as well. The work provides a new, publicly available Vietnamese OCR-VQA resource, evaluates strong baselines, and outlines future directions leveraging larger vision-language models and alternative OCR systems for improved scene-text understanding in Vietnamese.

Abstract

Optical Character Recognition - Visual Question Answering (OCR-VQA) is the task of answering text information contained in images that have just been significantly developed in the English language in recent years. However, there are limited studies of this task in low-resource languages such as Vietnamese. To this end, we introduce a novel dataset, ViOCRVQA (Vietnamese Optical Character Recognition - Visual Question Answering dataset), consisting of 28,000+ images and 120,000+ question-answer pairs. In this dataset, all the images contain text and questions about the information relevant to the text in the images. We deploy ideas from state-of-the-art methods proposed for English to conduct experiments on our dataset, revealing the challenges and difficulties inherent in a Vietnamese dataset. Furthermore, we introduce a novel approach, called VisionReader, which achieved 0.4116 in EM and 0.6990 in the F1-score on the test set. Through the results, we found that the OCR system plays a very important role in VQA models on the ViOCRVQA dataset. In addition, the objects in the image also play a role in improving model performance. We open access to our dataset at link (https://github.com/qhnhynmm/ViOCRVQA.git) for further research in OCR-VQA task in Vietnamese.

ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images

TL;DR

ViOCRVQA addresses the underexplored OCR-VQA problem for Vietnamese by introducing the ViOCRVQA dataset (28,282 images with 123,781 QA pairs) and a multimodal VisionReader model. VisionReader fuses textual OCR tokens, object features from VinVL, and grid features from ViT within a transformer encoder-decoder to generate exact-text answers from book-cover images. The paper demonstrates that OCR quality and object context substantially influence performance, with VisionReader (ViT5 backbone) achieving the strongest EM of 41.16% and F1 of 69.90% on the test set, and significant gains in book-genre prediction as well. The work provides a new, publicly available Vietnamese OCR-VQA resource, evaluates strong baselines, and outlines future directions leveraging larger vision-language models and alternative OCR systems for improved scene-text understanding in Vietnamese.

Abstract

Optical Character Recognition - Visual Question Answering (OCR-VQA) is the task of answering text information contained in images that have just been significantly developed in the English language in recent years. However, there are limited studies of this task in low-resource languages such as Vietnamese. To this end, we introduce a novel dataset, ViOCRVQA (Vietnamese Optical Character Recognition - Visual Question Answering dataset), consisting of 28,000+ images and 120,000+ question-answer pairs. In this dataset, all the images contain text and questions about the information relevant to the text in the images. We deploy ideas from state-of-the-art methods proposed for English to conduct experiments on our dataset, revealing the challenges and difficulties inherent in a Vietnamese dataset. Furthermore, we introduce a novel approach, called VisionReader, which achieved 0.4116 in EM and 0.6990 in the F1-score on the test set. Through the results, we found that the OCR system plays a very important role in VQA models on the ViOCRVQA dataset. In addition, the objects in the image also play a role in improving model performance. We open access to our dataset at link (https://github.com/qhnhynmm/ViOCRVQA.git) for further research in OCR-VQA task in Vietnamese.
Paper Structure (32 sections, 14 equations, 19 figures, 14 tables)

This paper contains 32 sections, 14 equations, 19 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Well-known Visual Question Answering Datasets
  4. Visual Question Answering Datasets in Vietnamese
  5. Visual Question Answering Methods
  6. Dataset Creation
  7. Collecting images and information
  8. Data Cleaning
  9. Creating question templates
  10. Statistics
  11. Dataset Comparison
  12. Linguistic Level
  13. Human Evaluation
  14. Methodology
  15. Multimodal Features Embedding
  16. ...and 17 more sections

Figures (19)

  • Figure 1: Several examples from the ViOCRVQA dataset.
  • Figure 2: The construction process of the ViOCRVQA dataset.
  • Figure 3: Distribution ViOCRVQA and OCR-VQA-200K
  • Figure 4: Overview of VisionReader structure.
  • Figure 5: Several examples of question-answer pairs related to title, title with unusual fonts, and messy arrangements.
  • ...and 14 more figures