Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VietMEAgent: Culturally-Aware Few-Shot Multimodal Explanation for Vietnamese Visual Question Answering

Hai-Dang Nguyen, Minh-Anh Dang, Minh-Tan Le, Minh-Tuan Le

TL;DR

The paper addresses the challenge of culturally grounded VQA for Vietnamese content and the need for interpretable reasoning. It proposes VietMEAgent, a four-stage framework that combines a cultural object detector, a few-shot program generator, a Vietnamese cultural knowledge base, and a multimodal explanation module to ground answers and rationales. A dataset of 28,484 images with 91,149 questions across 12 cultural categories supports evaluation and cultural coverage. Results show improvements in Cultural Accuracy and Explanation Quality over strong baselines, demonstrating that explicit cultural grounding and programmatic reasoning can enhance both performance and interpretable AI in cultural domains.

Abstract

Contemporary Visual Question Answering (VQA) systems remain constrained when confronted with culturally specific content, largely because cultural knowledge is under-represented in training corpora and the reasoning process is not rendered interpretable to end users. This paper introduces VietMEAgent, a multimodal explainable framework engineered for Vietnamese cultural understanding. The method integrates a cultural object detection backbone with a structured program generation layer, yielding a pipeline in which answer prediction and explanation are tightly coupled. A curated knowledge base of Vietnamese cultural entities serves as an explicit source of background information, while a dual-modality explanation module combines attention-based visual evidence with structured, human-readable textual rationales. We further construct a Vietnamese Cultural VQA dataset sourced from public repositories and use it to demonstrate the practicality of programming-based methodologies for cultural AI. The resulting system provides transparent explanations that disclose both the computational rationale and the underlying cultural context, supporting education and cultural preservation with an emphasis on interpretability and cultural sensitivity.

VietMEAgent: Culturally-Aware Few-Shot Multimodal Explanation for Vietnamese Visual Question Answering

TL;DR

The paper addresses the challenge of culturally grounded VQA for Vietnamese content and the need for interpretable reasoning. It proposes VietMEAgent, a four-stage framework that combines a cultural object detector, a few-shot program generator, a Vietnamese cultural knowledge base, and a multimodal explanation module to ground answers and rationales. A dataset of 28,484 images with 91,149 questions across 12 cultural categories supports evaluation and cultural coverage. Results show improvements in Cultural Accuracy and Explanation Quality over strong baselines, demonstrating that explicit cultural grounding and programmatic reasoning can enhance both performance and interpretable AI in cultural domains.

Abstract

Contemporary Visual Question Answering (VQA) systems remain constrained when confronted with culturally specific content, largely because cultural knowledge is under-represented in training corpora and the reasoning process is not rendered interpretable to end users. This paper introduces VietMEAgent, a multimodal explainable framework engineered for Vietnamese cultural understanding. The method integrates a cultural object detection backbone with a structured program generation layer, yielding a pipeline in which answer prediction and explanation are tightly coupled. A curated knowledge base of Vietnamese cultural entities serves as an explicit source of background information, while a dual-modality explanation module combines attention-based visual evidence with structured, human-readable textual rationales. We further construct a Vietnamese Cultural VQA dataset sourced from public repositories and use it to demonstrate the practicality of programming-based methodologies for cultural AI. The resulting system provides transparent explanations that disclose both the computational rationale and the underlying cultural context, supporting education and cultural preservation with an emphasis on interpretability and cultural sensitivity.

Paper Structure

This paper contains 20 sections, 3 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Textual Explanation Methodologies for VQA
  4. Multimodal Explanation Frameworks for VQA
  5. Few-Shot Explanation Methodologies for VQA
  6. METHODOLOGY
  7. System Architecture
  8. Cultural Object Detection Framework
  9. Program Generation and Execution
  10. Multimodal Explanation Generation
  11. VIETNAMESE CULTURAL VQA DATASET
  12. Construction and Coverage
  13. Quality Assessment and Statistics
  14. EXPERIMENTAL EVALUATION
  15. Setup and Metrics
  16. ...and 5 more sections

Figures (3)

  • Figure 1: VietMEAgent producing multimodal explanations that align visual evidence (e.g., attention heatmaps and bounding boxes) with structured textual rationales. The example illustrates identification of Hạ Long Bay accompanied by contextual cultural information.
  • Figure 2: Overall architecture. Cultural object detection feeds a program generator that compiles questions into executable steps. Program execution integrates a Vietnamese cultural knowledge base, and an explanation module renders aligned visual and textual rationales.
  • Figure 3: Performance comparison across BLEU-4, Cultural Accuracy, and Explanation Quality. VietMEAgent improves both task performance and transparency relative to strong baselines.