Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

Maurits Bleeker, Mariya Hendriksen, Andrew Yates, Maarten de Rijke

TL;DR

This work probes shortcut learning in vision-language contrastive learning when images have multiple captions, introducing SVL, a controlled framework that injects synthetic shortcuts into image-caption tuples to quantify reliance on shortcut cues. It analyzes the suboptimality of InfoNCE in producing task-optimal representations under multi-caption settings and proposes two mitigation strategies, Latent Target Decoding (LTD) and Implicit Feature Modification (IFM). Empirical results on CLIP and VSE++ across Flickr30k and MS-COCO show that models predominantly rely on shortcuts and shared features, with LTD and IFM offering partial improvement but not fully eliminating shortcut-driven behavior. The paper highlights fundamental limits of current contrastive VL objectives for achieving task-optimal representations and provides a practical benchmarking framework and actionable reduction techniques for future work.

Abstract

Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or whether the contrastive learning setup encourages the learning of a simple shortcut that minimizes contrastive loss. We introduce synthetic shortcuts for vision-language: a training and evaluation framework where we inject synthetic shortcuts into image-text data. We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features that represent the shortcut. Hence, contrastive losses are not sufficient to learn task-optimal representations, i.e., representations that contain all task-relevant information shared between the image and associated captions. We examine two methods to reduce shortcut learning in our training and evaluation framework: (i) latent target decoding and (ii) implicit feature modification. We show empirically that both methods improve performance on the evaluation task, but only partly reduce shortcut learning when training and evaluating with our shortcut learning framework. Hence, we show the difficulty and challenge of our shortcut learning framework for contrastive vision-language representation learning.

Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

TL;DR

This work probes shortcut learning in vision-language contrastive learning when images have multiple captions, introducing SVL, a controlled framework that injects synthetic shortcuts into image-caption tuples to quantify reliance on shortcut cues. It analyzes the suboptimality of InfoNCE in producing task-optimal representations under multi-caption settings and proposes two mitigation strategies, Latent Target Decoding (LTD) and Implicit Feature Modification (IFM). Empirical results on CLIP and VSE++ across Flickr30k and MS-COCO show that models predominantly rely on shortcuts and shared features, with LTD and IFM offering partial improvement but not fully eliminating shortcut-driven behavior. The paper highlights fundamental limits of current contrastive VL objectives for achieving task-optimal representations and provides a practical benchmarking framework and actionable reduction techniques for future work.

Abstract

Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or whether the contrastive learning setup encourages the learning of a simple shortcut that minimizes contrastive loss. We introduce synthetic shortcuts for vision-language: a training and evaluation framework where we inject synthetic shortcuts into image-text data. We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features that represent the shortcut. Hence, contrastive losses are not sufficient to learn task-optimal representations, i.e., representations that contain all task-relevant information shared between the image and associated captions. We examine two methods to reduce shortcut learning in our training and evaluation framework: (i) latent target decoding and (ii) implicit feature modification. We show empirically that both methods improve performance on the evaluation task, but only partly reduce shortcut learning when training and evaluating with our shortcut learning framework. Hence, we show the difficulty and challenge of our shortcut learning framework for contrastive vision-language representation learning.
Paper Structure (33 sections, 2 theorems, 14 equations, 6 figures, 3 tables)

This paper contains 33 sections, 2 theorems, 14 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Analysis
  3. Preliminaries
  4. Analysis of Contrastive Vision-Language Representation Learning for Multiple Captions per Image
  5. Synthetic Shortcuts to Control Shared Information
  6. Synthetic Shortcuts and their Impact on Learned Representations and Evaluation Performance
  7. Findings
  8. Upshot
  9. Reducing Shortcut Learning
  10. Latent Target Decoding
  11. Implicit Feature Modification
  12. Method Comparison
  13. Experimental Results
  14. Does Latent Target Decoding Reduce Shortcut Learning?
  15. Does Implicit Feature Modification Reduce Shortcut Learning?
  16. ...and 18 more sections

Key Result

Theorem 1

Given an image $\mathbf{x}_{\mathcal{I}}^{}{}$, a set of matching captions $\mathcal{C} = \{ \mathbf{x}_{\mathcal{C}_{A}}^{}, \mathbf{x}_{\mathcal{C}_{B}}^{} \}$, and a contrastive learning loss function $\mathcal{L}_{\text{InfoNCE}}^{}$ that optimizes for task $T$, image representations learned dur

Figures (6)

  • Figure 1: Shared vs. caption-specific information given an example of one image and two associated captions $\mathbf{x}_{\mathcal{C}_{A}}^{}$ and $\mathbf{x}_{\mathcal{C}_{B}}^{}$. The purple color indicates information shared between the image and both captions. The green color indicates task-relevant information specific for $\mathbf{x}_{\mathcal{C}_{A}}^{}$. The blue color indicates task-relevant information specific for $\mathbf{x}_{\mathcal{C}_{B}}^{}$.
  • Figure 2: Synthetic shortcuts in the context of minimal shared and task-optimal information for vision-language representation learning with multiple captions per image. The purple color represents features shared among the image and all captions (minimal shared information). The yellow color represents caption-specific features (unique information). The grey color indicates features that are not present in both the image and any of the captions (task-irrelevant information). The red color indicates synthetic shortcuts. We demonstrate that while shortcuts exist in both scenarios, minimal shared information also includes information shared among the image and all associated captions, whereas task-optimal information combines both minimal shared information and caption-specific information.
  • Figure 3: We define $H(\mathbf{x}_{\mathcal{I}}^{}{})$ as image information, $H(\mathbf{x}_{\mathcal{C}_{A}}^{})$ and $H(\mathbf{x}_{\mathcal{C}_{B}}^{})$ as caption information; both captions only describe the information depicted in the image and contain shared and caption-specific information. We further define $C_A = I( \mathbf{x}_{\mathcal{I}}^{}{} ; \mathbf{x}_{\mathcal{C}_{A}}^{} \mid \mathbf{x}_{\mathcal{C}_{B}}^{})$ and $C_B = I( \mathbf{x}_{\mathcal{I}}^{}{} ; \mathbf{x}_{\mathcal{C}_{B}}^{} \mid \mathbf{x}_{\mathcal{C}_{A}}^{})$ as caption-specific information; $S = I(\mathbf{x}_{\mathcal{I}}^{}{} ; \mathbf{x}_{\mathcal{C}_{A}}^{} ; \mathbf{x}_{\mathcal{C}_{B}}^{})$ as shared information; $\neg R = H( \mathbf{x}_{\mathcal{I}}^{}{} \mid \mathbf{x}_{\mathcal{C}_{A}}^{} , \mathbf{x}_{\mathcal{C}_{B}}^{} )$ as task-irrelevant information; $R = C_A + C_B + S$ as task-relevant information.
  • Figure 4: An image-caption pair from the MS-COCO dataset with a shortcut added to both the image and the caption.
  • Figure 5: Effect of synthetic shortcuts on CLIP and VSE++ performance on ICR task. The dotted line represents the maximum achievable recall sum, while the dashed line for CLIP indicates its zero-shot evaluation performance (Best viewed in color.)
  • ...and 1 more figures

Theorems & Definitions (7)

  • Definition 2.1: Sufficient caption representation
  • Definition 2.2: Sufficient image representation
  • Definition 2.3: Minimally sufficient image representation
  • Definition 2.4: Task-optimal image representation
  • Theorem 1: Suboptimality of contrastive learning with multiple captions per image
  • Theorem 1: Suboptimality of contrastive learning with multiple captions per image
  • proof