Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Updating CLIP to Prefer Descriptions Over Captions

Amir Zur, Elisa Kreiss, Karel D'Oosterlinck, Christopher Potts, Atticus Geiger

TL;DR

This work updates the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability.

Abstract

Although CLIPScore is a powerful generic metric that captures the similarity between a text and an image, it fails to distinguish between a caption that is meant to complement the information in an image and a description that is meant to replace an image entirely, e.g., for accessibility. We address this shortcoming by updating the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability. This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction.

Updating CLIP to Prefer Descriptions Over Captions

TL;DR

This work updates the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability.

Abstract

Although CLIPScore is a powerful generic metric that captures the similarity between a text and an image, it fails to distinguish between a caption that is meant to complement the information in an image and a description that is meant to replace an image entirely, e.g., for accessibility. We address this shortcoming by updating the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability. This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction.
Paper Structure (35 sections, 7 equations, 4 figures, 2 tables)

This paper contains 35 sections, 7 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Accessibility
  4. Referenceless Text-Image Evaluation Metrics
  5. Methods
  6. Behavioral Objective
  7. IIT-DAS Objective
  8. Fine-tuning
  9. Experiments
  10. Concadia and Transfer Evaluations
  11. Results and Discussion
  12. BLV and Sighted Human Evaluations
  13. Overall
  14. Imaginability
  15. Relevance
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Visualization of a single training step on Concadia, updating CLIP to prefer descriptions to captions. Though both objectives update CLIP to be sensitive to the description--caption distinction, IIT-DAS localizes this distinction to a subspace of CLIP's activations.
  • Figure 2: Mediated integrated gradient results.
  • Figure 3: Accuracy on the Concadia test set, and the three transfer tasks selected for transfer evaluation (CIFAR-100, Food101, and ImageNet).
  • Figure 4: Transfer score (averaged recovery percentage over all transfer tasks) over accuracy on the Concadia test set. A point on a seeded run yields a trade-off between sensitivity to the caption--description distinction and preserving the capabilities of CLIP. The joint objective refers to a training run minimizing both the behavioral and the IIT-DAS objective (see Appendix \ref{['app:training-details']}).