SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations

TL;DR

SugarCrepe++ introduces a multimodal, lexically diverse dataset to probe how vision-language and unimodal language models disentangle semantics from lexical variation. The dataset augments the prior SugarCrepe with a second semantically identical caption (P2) that differs lexically, paired with a hard negative (N), enabling a 3-way semantic (in)equivalence evaluation in both image-to-text and text-to-text modes. Comprehensive benchmarking shows that VLMs struggle to distinguish semantic from lexical changes, with performance gaps widening for attribute/object swaps and relation replacements; text encoders emerge as a bottleneck, though multi-objective pretraining and compositionality methods help but do not close the gap to human performance. Unimodal LMs, while sometimes outperforming VLMs in TOT, still show strong lexical sensitivity and large cross-subset variance, underscoring the need for further advances in semantic grounding and compositionality. SugarCrepe++ thus provides a rigorous, generalizable challenge for the vision-language community to develop models with a more robust understanding of semantics independent of lexical form.

Abstract

Despite their remarkable successes, state-of-the-art large language models (LLMs), including vision-and-language models (VLMs) and unimodal language models (ULMs), fail to understand precise semantics. For example, semantically equivalent sentences expressed using different lexical compositions elicit diverging representations. The degree of this divergence and its impact on encoded semantics is not very well understood. In this paper, we introduce the SUGARCREPE++ dataset to analyze the sensitivity of VLMs and ULMs to lexical and semantic alterations. Each sample in SUGARCREPE++ dataset consists of an image and a corresponding triplet of captions: a pair of semantically equivalent but lexically different positive captions and one hard negative caption. This poses a 3-way semantic (in)equivalence problem to the language models. We comprehensively evaluate VLMs and ULMs that differ in architecture, pre-training objectives and datasets to benchmark the performance of SUGARCREPE++ dataset. Experimental results highlight the difficulties of VLMs in distinguishing between lexical and semantic variations, particularly in object attributes and spatial relations. Although VLMs with larger pre-training datasets, model sizes, and multiple pre-training objectives achieve better performance on SUGARCREPE++, there is a significant opportunity for improvement. We show that all the models which achieve better performance on compositionality datasets need not perform equally well on SUGARCREPE++, signifying that compositionality alone may not be sufficient for understanding semantic and lexical alterations. Given the importance of the property that the SUGARCREPE++ dataset targets, it serves as a new challenge to the vision-and-language community.

Figures (10)

• Figure 1: Examples from SugarCrepe++ (SC++) dataset. $P_1$ and $P_2$ are semantically equivalent but lexically different while $N$ is semantically different than both $P_1$ and $P_2$ despite its lexical similarity with $P_1$. The adjacent line charts highlight the performance gaps in VLMs discovered upon re-evaluation using SC++, and shows that strong lexical and semantic understanding may not be required to achieve better performance on SugarCrepe (SC).
• Figure 2: Role playing prompt for "Data Generator AI".
• Figure 3: Rules and demonstration sub-prompts used to condition the generator LLM.
• Figure 4: Validation Meta-prompt used to validate the consistency of the generated caption and original caption. We use the isConsistent output to signal the regeneration of a semantically inconsistent caption.
• Figure 5: Examples of common errors in the LLM generated positive sentences (P$_2$). We provide five of the most common types of errors. Manual correction refers to the corrected sentences after the human validation step. Expert human annotators carefully checked each output of the LLM and corrected the erroneous sentences to maintain grammatical and semantic equivalence with the original positive caption (P$_1$)