See or Guess: Counterfactually Regularized Image Captioning

TL;DR

A generic image captioning framework that employs causal inference to make existing models more capable of interventional tasks, and counterfactually explainable, and enables models to handle counterfactual scenarios, increasing their generalizability.

Abstract

Image captioning, which generates natural language descriptions of the visual information in an image, is a crucial task in vision-language research. Previous models have typically addressed this task by aligning the generative capabilities of machines with human intelligence through statistical fitting of existing datasets. While effective for normal images, they may struggle to accurately describe those where certain parts of the image are obscured or edited, unlike humans who excel in such cases. These weaknesses they exhibit, including hallucinations and limited interpretability, often hinder performance in scenarios with shifted association patterns. In this paper, we present a generic image captioning framework that employs causal inference to make existing models more capable of interventional tasks, and counterfactually explainable. Our approach includes two variants leveraging either total effect or natural direct effect. Integrating them into the training process enables models to handle counterfactual scenarios, increasing their generalizability. Extensive experiments on various datasets show that our method effectively reduces hallucinations and improves the model's faithfulness to images, demonstrating high portability across both small-scale and large-scale image-to-text models. The code is available at https://github.com/Aman-4-Real/See-or-Guess.

Figures (9)

• Figure 1: An example of generated captions of different methods in the factual and two counterfactual scenarios.
• Figure 2: Illustration of causal graphs and counterfactual causal effect notations.
• Figure 3: Our framework of counterfactual regularization. (a) shows how to prepare counterfactual images and captions by example. (b) illustrates how the TE loss and NDE loss are calculated in the example. Counterfactual captions are in blue. The phrase corresponding to the image region in the mask is "black poodle". Best viewed in color.
• Figure 4: Examples of generated captions by different methods on some masked or inpainted counterfactual images. Phrases highlighted in red are hallucinations that do not exist in the counterfactual image.
• Figure 5: Illustration of interpretability evaluation.