Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, Kyle Mahowald

TL;DR

This work interrogates why Winoground challenges state-of-the-art visuolinguistic models by introducing a fine-grained tag taxonomy, textual variants, and a suite of probing analyses. It shows that failures are not solely due to linguistic composition but largely stem from difficulties in fusing visual and textual representations, with substantial variability across model families and item categories. The authors demonstrate that even augmented text variants with separable semantics do not readily improve cross-modal matching, and they emphasize evaluating models with per-tag breakdowns to better understand capabilities and limitations. The study provides a comprehensive framework and resources to diagnose and improve multimodal grounding and compositionality in large pretrained transformers.

Abstract

Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .

Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

TL;DR

This work interrogates why Winoground challenges state-of-the-art visuolinguistic models by introducing a fine-grained tag taxonomy, textual variants, and a suite of probing analyses. It shows that failures are not solely due to linguistic composition but largely stem from difficulties in fusing visual and textual representations, with substantial variability across model families and item categories. The authors demonstrate that even augmented text variants with separable semantics do not readily improve cross-modal matching, and they emphasize evaluating models with per-tag breakdowns to better understand capabilities and limitations. The study provides a comprehensive framework and resources to diagnose and improve multimodal grounding and compositionality in large pretrained transformers.

Abstract

Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .
Paper Structure (37 sections, 3 equations, 3 figures, 5 tables)

This paper contains 37 sections, 3 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Relaxing Winoground Constraints
  4. Recall at $k>1$
  5. Setting
  6. Methods
  7. Results
  8. Task Adaptation
  9. Methods
  10. Results
  11. Characterizing the Challenges Presented by Winoground Items
  12. Potentially Easy Pairs
  13. NonCompositional
  14. Potentially Difficult Pairs: In-Domain
  15. AmbiguouslyCorrect
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Extending the (A) original Winoground task, which presents a minimal semantic pair of image captions and their corresponding images, we (B) create new fine-grained tags, identify their main challenges, and evaluate performance separately on each subcategory (Section \ref{['sec:taxonomy']}); further, we (C) also create textual variants of the original captions where they are no longer minimal semantic pairs. Models are still unable to succeed on the Winoground task (Section \ref{['sec:augmentations']}) when given such linearly separable pairs.
  • Figure 2: A taxonomy of Winoground schemes, with scores on CLIP in the bottom row for Text/Image/Group score respectively and with above-chance performance in bold.
  • Figure 3: Top: Average SVC Probe Margin Width Across Model Layers for LXMERT, UNITER, and CLIP. Bottom: Test Set Accuracy of a 4-layer text-only probe over [CLS] Embeddings Across Model Layers for LXMERT, UNITER, and CLIP.