Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Role of Visual Grounding in VQA

Daniel Reich, Tanja Schultz

TL;DR

By consolidating fundamental insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC exploitation in OOD testing, which explains why the relationship between VG and OOD accuracy has been difficult to define.

Abstract

Visual Grounding (VG) in VQA refers to a model's proclivity to infer answers based on question-relevant image regions. Conceptually, VG identifies as an axiomatic requirement of the VQA task. In practice, however, DNN-based VQA models are notorious for bypassing VG by way of shortcut (SC) learning without suffering obvious performance losses in standard benchmarks. To uncover the impact of SC learning, Out-of-Distribution (OOD) tests have been proposed that expose a lack of VG with low accuracy. These tests have since been at the center of VG research and served as basis for various investigations into VG's impact on accuracy. However, the role of VG in VQA still remains not fully understood and has not yet been properly formalized. In this work, we seek to clarify VG's role in VQA by formalizing it on a conceptual level. We propose a novel theoretical framework called "Visually Grounded Reasoning" (VGR) that uses the concepts of VG and Reasoning to describe VQA inference in ideal OOD testing. By consolidating fundamental insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC exploitation in OOD testing, which explains why the relationship between VG and OOD accuracy has been difficult to define. Finally, we propose an approach to create OOD tests that properly emphasize a requirement for VG, and show how to improve performance on them.

On the Role of Visual Grounding in VQA

TL;DR

By consolidating fundamental insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC exploitation in OOD testing, which explains why the relationship between VG and OOD accuracy has been difficult to define.

Abstract

Visual Grounding (VG) in VQA refers to a model's proclivity to infer answers based on question-relevant image regions. Conceptually, VG identifies as an axiomatic requirement of the VQA task. In practice, however, DNN-based VQA models are notorious for bypassing VG by way of shortcut (SC) learning without suffering obvious performance losses in standard benchmarks. To uncover the impact of SC learning, Out-of-Distribution (OOD) tests have been proposed that expose a lack of VG with low accuracy. These tests have since been at the center of VG research and served as basis for various investigations into VG's impact on accuracy. However, the role of VG in VQA still remains not fully understood and has not yet been properly formalized. In this work, we seek to clarify VG's role in VQA by formalizing it on a conceptual level. We propose a novel theoretical framework called "Visually Grounded Reasoning" (VGR) that uses the concepts of VG and Reasoning to describe VQA inference in ideal OOD testing. By consolidating fundamental insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC exploitation in OOD testing, which explains why the relationship between VG and OOD accuracy has been difficult to define. Finally, we propose an approach to create OOD tests that properly emphasize a requirement for VG, and show how to improve performance on them.

Paper Structure

This paper contains 53 sections, 6 equations, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Visually Grounded Reasoning
  4. Reasoning
  5. Hypothesis 1.
  6. Visual Grounding
  7. Hypothesis 2.
  8. The VGR Proposition
  9. VGR Proposition.
  10. Hypothesis 1 & 2 are insufficient as individual descriptions of SC-free inference
  11. Summary of Insights
  12. VGR in Practice
  13. VGR corollaries of SC-free inference
  14. Corollary 1: BGC is zero.
  15. Corollary 2: GGC equals Accuracy.
  16. ...and 38 more sections

Figures (2)

  • Figure 1: Examples of symbolic visual features and samples in GQA-AUG. Left: Symbolic feature representation. AUG-ID (middle) shows an original test sample (question and visual features) with the ground-truth answer ("cat"). AUG-OOD (right) contains new samples that differ in both answer ("dog", "bird", ...) and feature content (modified to support the answer). The question is not changed.
  • Figure 2: Illustration of AUG-OOD results for DET and INF-trained models from Table \ref{['table:INF_AUG_allmodels']} (averages over five differently seeded runs). Discussion in Sec. \ref{['ch8_subsec:learning_ir_reasoning_result_discussion']}.