Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MaXM: Towards Multilingual Visual Question Answering

Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V. Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, Radu Soricut

TL;DR

This work tackles multilingual visual question answering by introducing a scalable translation-based data-generation pipeline (TransVQ2A) that converts multilingual captions into English QA pairs and back, augmented by DirectQG for coverage gaps. It builds MaXM, a test-only VQA benchmark in 7 diverse languages using XM3600 captions, with a rigorous human-in-the-loop annotation workflow and analysis. The paper also presents Simple MPT, a lightweight open-ended multilingual VQA model, and benchmarks it against state-of-the-art English and multilingual VQA systems, showing substantial room for improvement but strong multilingual capabilities. Overall, MaXM provides a practical, out-of-domain evaluation resource to spur advancement in truly multilingual VQA from data creation to modeling.

Abstract

Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA.

MaXM: Towards Multilingual Visual Question Answering

TL;DR

This work tackles multilingual visual question answering by introducing a scalable translation-based data-generation pipeline (TransVQ2A) that converts multilingual captions into English QA pairs and back, augmented by DirectQG for coverage gaps. It builds MaXM, a test-only VQA benchmark in 7 diverse languages using XM3600 captions, with a rigorous human-in-the-loop annotation workflow and analysis. The paper also presents Simple MPT, a lightweight open-ended multilingual VQA model, and benchmarks it against state-of-the-art English and multilingual VQA systems, showing substantial room for improvement but strong multilingual capabilities. Overall, MaXM provides a practical, out-of-domain evaluation resource to spur advancement in truly multilingual VQA from data creation to modeling.

Abstract

Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA.
Paper Structure (24 sections, 12 figures, 9 tables)

This paper contains 24 sections, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. VQA and Multilingual Multimodal Benchmarks
  4. VQA Data Creation
  5. Multilingual VQA Data Creation
  6. Background: $\mathrm{VQ}^{2}\!\mathrm{A}$
  7. Translation-based $\mathrm{VQ}^{2}\!\mathrm{A}$ ($\mathrm{TransVQ}^{2}\!\mathrm{A}$)
  8. Direct Question Generation (DirectQG)
  9. MaXM: Multilingual VQA Benchmark
  10. Data Sources
  11. Large-Scale mVQA Data Creation
  12. Human Annotation
  13. Analysis and Discussion
  14. Evaluation
  15. Evaluation Protocol
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Multilingual VQA Data in 7 languages. The data is automatically generated from multilingual captions and then verified and adjusted by humans. From top to bottom: English (en), French (fr), Hindi (hi), Hebrew (iw), Romanian (ro), Thai (th), and Chinese (zh).
  • Figure 2: Our approach to multilingual VQA data generation, which is easy to scale, highly automatic and only requiring humans to modify "Almost Correct" questions or correct/expand answers (left) or filter out "Incorrect" questions(right). MT is short for automatic machine translation.
  • Figure 3: The diversity of multilingual captions in $\mathrm{XM3600}$. We show the captions (their English translations) from 4 languages for the images of a snow cannon (left) and xiao long bao (right).
  • Figure 4: Top answer cloud is for "What" questions (excluding "What color"). Bottom answer clouds from left to right are for "What color", Boolean "Is/Are/Was/Were/Do/Does/Did", and "How many" questions, respectively.
  • Figure 5: Detailed Instructions on Question Annotation
  • ...and 7 more figures