Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions

Shun Inadumi, Seiya Kawano, Akishige Yuguchi, Yasutomo Kawanishi, Koichiro Yoshino

TL;DR

This work tackles ambiguity in visual question answering, with a focus on Japanese where subject/object ellipses and directives complicate comprehension. It introduces GazeVQA, a dataset of gaze-grounded questions, and a model ClipCap + Adapter that integrates gaze-target RoIs derived from a gaze estimator. Experiments show that grounding with gaze information can improve VQA accuracy in some settings, especially when adapters are trained while full end-to-end fine-tuning is limited. The results highlight both the potential and limitations of gaze-grounded grounding for resolving linguistic ambiguity in vision-and-language tasks, with implications for human-robot interaction and multilingual VQA systems.

Abstract

Situated conversations, which refer to visual information as visual question answering (VQA), often contain ambiguities caused by reliance on directive information. This problem is exacerbated because some languages, such as Japanese, often omit subjective or objective terms. Such ambiguities in questions are often clarified by the contexts in conversational situations, such as joint attention with a user or user gaze information. In this study, we propose the Gaze-grounded VQA dataset (GazeVQA) that clarifies ambiguous questions using gaze information by focusing on a clarification process complemented by gaze information. We also propose a method that utilizes gaze target estimation results to improve the accuracy of GazeVQA tasks. Our experimental results showed that the proposed method improved the performance in some cases of a VQA system on GazeVQA and identified some typical problems of GazeVQA tasks that need to be improved.

A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions

TL;DR

This work tackles ambiguity in visual question answering, with a focus on Japanese where subject/object ellipses and directives complicate comprehension. It introduces GazeVQA, a dataset of gaze-grounded questions, and a model ClipCap + Adapter that integrates gaze-target RoIs derived from a gaze estimator. Experiments show that grounding with gaze information can improve VQA accuracy in some settings, especially when adapters are trained while full end-to-end fine-tuning is limited. The results highlight both the potential and limitations of gaze-grounded grounding for resolving linguistic ambiguity in vision-and-language tasks, with implications for human-robot interaction and multilingual VQA systems.

Abstract

Situated conversations, which refer to visual information as visual question answering (VQA), often contain ambiguities caused by reliance on directive information. This problem is exacerbated because some languages, such as Japanese, often omit subjective or objective terms. Such ambiguities in questions are often clarified by the contexts in conversational situations, such as joint attention with a user or user gaze information. In this study, we propose the Gaze-grounded VQA dataset (GazeVQA) that clarifies ambiguous questions using gaze information by focusing on a clarification process complemented by gaze information. We also propose a method that utilizes gaze target estimation results to improve the accuracy of GazeVQA tasks. Our experimental results showed that the proposed method improved the performance in some cases of a VQA system on GazeVQA and identified some typical problems of GazeVQA tasks that need to be improved.
Paper Structure (38 sections, 4 equations, 6 figures, 8 tables)

This paper contains 38 sections, 4 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Question Answering with Contextual Information
  4. Gaze Target Estimation
  5. Gaze-grounded VQA Dataset
  6. Task Setting
  7. Data Collection
  8. Step 1: Selection of images and gaze information:
  9. Step 2: Collecting questions and answers:
  10. Step 3: Screening of questions and answers:
  11. Step 4: Preparation for test-set:
  12. Example
  13. Statistics and Analysis
  14. Methodology
  15. Baseline Model: ClipCap
  16. ...and 23 more sections

Figures (6)

  • Figure 1: Examples of questions and answers for GazeVQA proposed in this research: Square brackets denote omitted gaze target names. Multiple target points are assigned that correspond to source points.
  • Figure 2: Data collection process of our Gaze-grounded VQA dataset
  • Figure 3: Examples of GazeVQA test-set: AQ and answers in bold denote ambiguous questions and answers obtained through Step 3. CQ denotes questions clarified by annotator's work. The original questions and answers are given in Japanese. We put English translation in the bottom. Words denoted by square brackets are supplements in translations; the terms are omitted in the original Japanese questions.
  • Figure 4: Left: Overview of proposed system Right: Details of Image Encoder architecture
  • Figure 5: Outputs for baseline and proposed models: AQ, CQ, and A are respectively ambiguous questions, clarified questions, and examples of correct answers. Bolded results are models that scored best among five attempts. $GT$ and $I_s$ are denoted by red and green boxes.
  • ...and 1 more figures