RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models

TL;DR

RoCOCO introduces a robustness benchmark for image-text matching by perturbing MS-COCO captions semantically and images visually. The authors demonstrate substantial performance drops across a broad set of state-of-the-art VL models when challenged with these variations, indicating weak generalization to subtle semantic and visual changes. They propose Semantic Contrastive (SC) and Visual Contrastive (VC) losses to encourage finer-grained semantic and visual understanding during embedding learning. The work provides a practical evaluation framework and actionable mitigation strategies, with datasets and code released to enable broader adoption and further research in robust cross-modal understanding.

Abstract

With the extensive use of vision-language models in various downstream tasks, evaluating their robustness is crucial. In this paper, we propose a benchmark for assessing the robustness of vision-language models. We believe that a robust model should properly understand both linguistic and visual semantics and be resilient to explicit variations. In pursuit of this goal, we create new variants of texts and images in the MS-COCO test set and re-evaluate the state-of-the-art (SOTA) models with the new data. Specifically, we alter the meaning of text by replacing a word, and generate visually altered images that maintain some visual context while introducing noticeable pixel changes through image mixing techniques.Our evaluations on the proposed benchmark reveal substantial performance degradation in many SOTA models (e.g., Image-to-Text Recall@1: 81.9\% $\rightarrow$ 48.4\% in BLIP, 66.1\% $\rightarrow$ 37.6\% in VSE$\infty$), with the models often favoring the altered texts/images over the original ones. This indicates the current vision-language models struggle with subtle changes and often fail to understand the overall context of texts and images. Based on these findings, we propose semantic contrastive loss and visual contrastive loss to learn more robust embedding. Datasets and code are available at {\url{https://github.com/pseulki/rococo}}.

Figures (13)

• Figure 1: Motivating Example. (a) When we add a new caption with only one word changed from "umbrella" to "gun", this new caption is preferred by the model (Image-to-text). Likewise, when we add a new image created by inserting an unrelated image to the original one, this new image is ranked as top 1 (Text-to-image). (b) Recall@1 scores of existing SOTAs decrease when tested with our RoCOCO benchmark.
• Figure 2: Our Benchmark. By adding newly created confusing texts and images into the existing test set, our proposed benchmarks stress-test the model's robustness in understanding visual and semantic details.
• Figure 3: Example of semantically altered captions. (Left) Original MS-COCO image and captions. (Right) Our generated captions, Rand-voca, Same-concept, Diff-concept, and Danger from top to bottom. The model's robustness is evaluated if it can correctly retrieve the original ground-truth caption, in the presence of newly generated semantically altered captions.
• Figure 4: Example of visually altered images with different $\lambda$.
• Figure 5: Examples of incorrectly retrieved texts with BLIP from Same-concept (Image-to-Text).Captions with incorrect details are ranked at the top 1, while the correct captions are ranked lower.