SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality

TL;DR

Vision-language benchmarks for compositionality have been shown to be hackable, with text-only baselines outperforming image-based models. The authors introduce SugarCrepe, a benchmark built with large-language-model-generated, fluent hard negatives and an adversarial refinement step to remove exploitable biases, enabling a more faithful evaluation. Reassessing recent compositionality techniques reveals that prior gains were largely overstated and often due to dataset artifacts rather than true compositional understanding. The work highlights the need for novel approaches beyond simple data augmentation and provides resources to replicate and extend the evaluation framework.

Abstract

In the last year alone, a surge of new benchmarks to measure compositional understanding of vision-language models have permeated the machine learning ecosystem. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. Surprisingly, we find significant biases in all these benchmarks rendering them hackable. This hackability is so dire that blind models with no access to the image outperform state-of-the-art vision-language models. To remedy this rampant vulnerability, we introduce SugarCrepe, a new benchmark for vision-language compositionality evaluation. We employ large language models, instead of rule-based templates used in previous benchmarks, to generate fluent and sensical hard negatives, and utilize an adversarial refinement mechanism to maximally reduce biases. We re-evaluate state-of-the-art models and recently proposed compositionality inducing strategies, and find that their improvements were hugely overestimated, suggesting that more innovation is needed in this important direction. We release SugarCrepe and the code for evaluation at: https://github.com/RAIVNLab/sugar-crepe.

Figures (9)

• Figure 1: Top row: We define Vera score gap as the score difference between the positive and hard negative texts: $\mathrm{Vera}(T^\mathrm{p}) - \mathrm{Vera}(T^\mathrm{n})$. The entire Vera score gap distribution lies on the positive spectrum, indicating that the template-generated hard negative texts usually have low plausibility. Bottom row: Similarly, Grammar score gap is defined by: $\mathrm{Grammar}(T^\mathrm{p}) - \mathrm{Grammar}(T^\mathrm{n})$. On grammar score, we also find that the distribution largely rests on the positive side, suggesting that most hard negative texts in existing benchmarks exhibit grammatical errors.
• Figure 2: Blind commonsense Vera model and Grammar model outperform state-of-the-art CLIP models on nearly all existing benchmarks by exploiting the nonsensical and non-fluent artifacts. This suggests that existing benchmarks are hackable and ineffective in measuring compositionality.
• Figure 3: Example prompt (black) and actual hard negative (green) generated from ChatGPT.
• Figure 4: We compare the Vera (top row) and Grammar (bottom row) score gap distributions between ARO+CREPE (leftmost column), SugarCrepe without adversarial refinement (middle), and SugarCrepe (rightmost). Top row: We see that Vera score gap distribution shifts from the positive spectrum to more centered around zero from ARO+CREPE to SugarCrepe without refinement. After adversarial refinement, we ensure the score gap distribution is centered around zero on SugarCrepe. Bottom row: Similarly, from ARO+CREPE to SugarCrepe, we see the Grammar score gap distribution shifts from the positive spectrum to centered around zero.
• Figure 5: We plot pretrained vision-language models' zero-shot top-1 accuracy on ImageNet versus their retrieval recall@1 on SugarCrepe, where $r$ is the Pearson correlation coefficient. This plot suggests that models' ImageNet zero-shot accuracy positively correlates with their compositionality.