Towards Deconfounded Image-Text Matching with Causal Inference

TL;DR

The paper tackles spurious correlations in image–text matching caused by intra‑ and inter‑modal biases and biased external knowledge. It introduces the Deconfounded Causal Inference Network (DCIN), which leverages Structural Causal Models and backdoor adjustment to learn causal relations and debias external knowledge through confounder dictionaries and debiased knowledge embeddings. DCIN comprises modules for global embeddings, confounder dictionaries, debiased knowledge, and a deconfounding mechanism that yields a joint deconfounded similarity score, optimized with a hinge loss. Experiments on Flickr30K and MSCOCO demonstrate improved generalization and state‑of‑the‑art or competitive performance, validating the effectiveness of removing spurious correlations and learning causal associations in cross‑modal retrieval.

Abstract

Prior image-text matching methods have shown remarkable performance on many benchmark datasets, but most of them overlook the bias in the dataset, which exists in intra-modal and inter-modal, and tend to learn the spurious correlations that extremely degrade the generalization ability of the model. Furthermore, these methods often incorporate biased external knowledge from large-scale datasets as prior knowledge into image-text matching model, which is inevitable to force model further learn biased associations. To address above limitations, this paper firstly utilizes Structural Causal Models (SCMs) to illustrate how intra- and inter-modal confounders damage the image-text matching. Then, we employ backdoor adjustment to propose an innovative Deconfounded Causal Inference Network (DCIN) for image-text matching task. DCIN (1) decomposes the intra- and inter-modal confounders and incorporates them into the encoding stage of visual and textual features, effectively eliminating the spurious correlations during image-text matching, and (2) uses causal inference to mitigate biases of external knowledge. Consequently, the model can learn causality instead of spurious correlations caused by dataset bias. Extensive experiments on two well-known benchmark datasets, i.e., Flickr30K and MSCOCO, demonstrate the superiority of our proposed method.

Figures (6)

• Figure 1: An illustration of spurious correlation in image-to-text retrieval. $D_1$, $D_2$ denote the visual and linguistic confounders, respectively. The probability represents the percentage of co-occurrence of two concepts in the training set, where ID denotes the visual concept. The top result is obtained by GPO, and the bottom is searched by our DCIN. Red and blue indicate wrong and correct words, respectively. Green boxes represent visual concepts that may have caused errors, with green words in the text corresponding to those visual concepts.
• Figure 2: Left: structural causal model (SCM) for image-text matching. Right: intervention with backdoor adjustment. The direction of an edge in the SCM indicate solely the causal relationship, directing from the cause to the effect. The variables $V$, $T$, $D_1$, $D_2$ refer to image, text, visual confounder and linguistic confounder, respectively.
• Figure 3: Illustration of the Deconfounded Causal Inference Network, which consists of three main modules designed to eliminate the spurious correlations between image and text features. (1) The Confounder Dictionary Embedding module constructs approximate visual and linguistic confounder dictionary.(2) The Debias Knowledge Embedding module incorporates debiased knowledge from the Visual Genome dataset to enhance the model's matching ability. (3) The Visual-Textual Feature Deconfound module uses causal inference to confront visual and linguistic confounders and eliminates the spurious correlations during image-text matching to obtain accurate image or text matching results.
• Figure 4: Left: structural causal model (SCM) for incorporating external knowledge. Right: intervention with backdoor adjustment. The variables $X$, $Y$ indicate semantic concepts, and $Z$ denotes confounder.
• Figure 5: The confidence level analysis for the approximate confounder dictionary on the Flickr30K test set.