Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation

Rhea Kapur, Robert Hawkins, Elisa Kreiss

TL;DR

This work tackles the problem that description length often masquerades as informativeness in image description evaluation. It introduces a contrast-set framework to define specificity and builds a dataset that decouples length from content, using 5,000 COCO images and multiple description variants. A CLIPScore-based metric operationalizes specificity via target-image discrimination against large distractor sets, and human studies validate that people prefer more specific descriptions regardless of length. The results demonstrate that length alone cannot explain specificity and that prompting strategies that allocate length toward discriminative content can improve specificity, advocating for evaluation methods that directly measure specificity rather than relying on verbosity as a proxy.

Abstract

Vision-language models (VLMs) are increasingly used to make visual content accessible via text-based descriptions. In current systems, however, description specificity is often conflated with their length. We argue that these two concepts must be disentangled: descriptions can be concise yet dense with information, or lengthy yet vacuous. We define specificity relative to a contrast set, where a description is more specific to the extent that it picks out the target image better than other possible images. We construct a dataset that controls for length while varying information content, and validate that people reliably prefer more specific descriptions regardless of length. We find that controlling for length alone cannot account for differences in specificity: how the length budget is allocated makes a difference. These results support evaluation approaches that directly prioritize specificity over verbosity.

When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation

TL;DR

This work tackles the problem that description length often masquerades as informativeness in image description evaluation. It introduces a contrast-set framework to define specificity and builds a dataset that decouples length from content, using 5,000 COCO images and multiple description variants. A CLIPScore-based metric operationalizes specificity via target-image discrimination against large distractor sets, and human studies validate that people prefer more specific descriptions regardless of length. The results demonstrate that length alone cannot explain specificity and that prompting strategies that allocate length toward discriminative content can improve specificity, advocating for evaluation methods that directly measure specificity rather than relying on verbosity as a proxy.

Abstract

Vision-language models (VLMs) are increasingly used to make visual content accessible via text-based descriptions. In current systems, however, description specificity is often conflated with their length. We argue that these two concepts must be disentangled: descriptions can be concise yet dense with information, or lengthy yet vacuous. We define specificity relative to a contrast set, where a description is more specific to the extent that it picks out the target image better than other possible images. We construct a dataset that controls for length while varying information content, and validate that people reliably prefer more specific descriptions regardless of length. We find that controlling for length alone cannot account for differences in specificity: how the length budget is allocated makes a difference. These results support evaluation approaches that directly prioritize specificity over verbosity.
Paper Structure (25 sections, 7 figures, 2 tables)

This paper contains 25 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Defining specificity via contrast sets
  4. Limitations of evaluation metrics
  5. Approach
  6. Dataset construction
  7. Measuring specificity
  8. Results
  9. Validating human specificity preferences
  10. Validating the specificity metric
  11. Distinguishing specificity from length
  12. Evaluating VLM length constraints
  13. Conclusion
  14. Prompts
  15. Verbose
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Pairwise human preferences by description type (95% bootstrapped CIs). Full results in \ref{['pref_details']}.
  • Figure 2: Cumulative distribution of ranks across description types relative to all other images.
  • Figure 3: Mean rank vs. description length by type. Point size: bin sample size; ribbons: 95% CIs; diamonds: overall means. Range excludes outliers.
  • Figure 4: Mean rank vs. description length by each type in the length-constrained case study (\ref{['ablations']}). Point size: bin sample size; ribbons: 95% CIs; diamonds: overall means. Range excludes outliers.
  • Figure 5: Experimental results of the preference study (in \ref{['human']} and \ref{['app:expmain']}) and the specificity study (in \ref{['app:expsuppl']}).
  • ...and 2 more figures