Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causal Debiasing for Visual Commonsense Reasoning

Jiayi Zou, Gengyun Jia, Bing-Kun Bao

TL;DR

This work addresses biases in Visual Commonsense Reasoning (VCR) by identifying co-occurrence and statistical biases that enable prediction shortcuts. It introduces VCR-OOD, text- and vision-focused out-of-distribution datasets, to evaluate generalization across modalities. The authors propose a causal debiasing framework based on backdoor adjustment using an answer dictionary and NWGM-style aggregation, complemented by a negative-sample loss to emphasize genuine visual content. Experiments show substantial OOD performance gains with limited impact on in-distribution results, offering a robust, scalable approach to debias multi-modal reasoning in VCR.

Abstract

Visual Commonsense Reasoning (VCR) refers to answering questions and providing explanations based on images. While existing methods achieve high prediction accuracy, they often overlook bias in datasets and lack debiasing strategies. In this paper, our analysis reveals co-occurrence and statistical biases in both textual and visual data. We introduce the VCR-OOD datasets, comprising VCR-OOD-QA and VCR-OOD-VA subsets, which are designed to evaluate the generalization capabilities of models across two modalities. Furthermore, we analyze the causal graphs and prediction shortcuts in VCR and adopt a backdoor adjustment method to remove bias. Specifically, we create a dictionary based on the set of correct answers to eliminate prediction shortcuts. Experiments demonstrate the effectiveness of our debiasing method across different datasets.

Causal Debiasing for Visual Commonsense Reasoning

TL;DR

This work addresses biases in Visual Commonsense Reasoning (VCR) by identifying co-occurrence and statistical biases that enable prediction shortcuts. It introduces VCR-OOD, text- and vision-focused out-of-distribution datasets, to evaluate generalization across modalities. The authors propose a causal debiasing framework based on backdoor adjustment using an answer dictionary and NWGM-style aggregation, complemented by a negative-sample loss to emphasize genuine visual content. Experiments show substantial OOD performance gains with limited impact on in-distribution results, offering a robust, scalable approach to debias multi-modal reasoning in VCR.

Abstract

Visual Commonsense Reasoning (VCR) refers to answering questions and providing explanations based on images. While existing methods achieve high prediction accuracy, they often overlook bias in datasets and lack debiasing strategies. In this paper, our analysis reveals co-occurrence and statistical biases in both textual and visual data. We introduce the VCR-OOD datasets, comprising VCR-OOD-QA and VCR-OOD-VA subsets, which are designed to evaluate the generalization capabilities of models across two modalities. Furthermore, we analyze the causal graphs and prediction shortcuts in VCR and adopt a backdoor adjustment method to remove bias. Specifically, we create a dictionary based on the set of correct answers to eliminate prediction shortcuts. Experiments demonstrate the effectiveness of our debiasing method across different datasets.
Paper Structure (8 sections, 5 equations, 3 figures, 5 tables)

This paper contains 8 sections, 5 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Construction of OOD datasets
  3. Our Method
  4. Experiments
  5. Experimental Setup
  6. Quantitative Result
  7. Ablation Study
  8. Conclusion

Figures (3)

  • Figure 1: (a) is the causal graph in Q$\rightarrow$A . (b) shows the confounder in Q$\rightarrow$A. (c) is our deconfounded in Q$\rightarrow$A. (d), (e) and (f) correspond to the representations in QA$\rightarrow$R respectively. The blue arrows indicate the prediction shortcuts.
  • Figure 2: The framework diagram of causal debiasing in the model. The blue arrows represent the prediction process of the backbone, and the orange arrows are the operations that introduce the do operator.
  • Figure 3: Performance changes with different parameter values ($\lambda$) on VCR-OOD-QA (a) and VCR-OOD-VA Datasets (b).