Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge

Brendan Park, Madeline Janecek, Naser Ezzati-Jivan, Yifeng Li, Ali Emami

TL;DR

WinoVis addresses the gap in multimodal common-sense reasoning by reframing the Winograd Schema Challenge for text-to-image models. It introduces a 500-scenario dataset generated with GPT-4 and an evaluation framework that isolates pronoun disambiguation from generic visual processing using DAAM heatmaps and IoU-based filtering. The approach reveals that even strong diffusion models like Stable Diffusion 2.0 achieve only modest pronoun-disambiguation performance (around $56.7\%$ precision) and that interpretability remains a major bottleneck, with SDXL showing notably weak heatmap reliability. The findings highlight concrete directions for improving multimodal pronoun resolution, including better entity separation, broader model diversity, bias analysis, enriched datasets, and more robust filtering techniques.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. However, applying this reasoning to multimodal domains, where understanding text and images together is essential, remains a substantial challenge. To address this, we introduce WinoVis, a novel dataset specifically designed to probe text-to-image models on pronoun disambiguation within multimodal contexts. Utilizing GPT-4 for prompt generation and Diffusion Attentive Attribution Maps (DAAM) for heatmap analysis, we propose a novel evaluation framework that isolates the models' ability in pronoun disambiguation from other visual processing challenges. Evaluation of successive model versions reveals that, despite incremental advancements, Stable Diffusion 2.0 achieves a precision of 56.7% on WinoVis, only marginally surpassing random guessing. Further error analysis identifies important areas for future research aimed at advancing text-to-image models in their ability to interpret and interact with the complex visual world.

Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge

TL;DR

WinoVis addresses the gap in multimodal common-sense reasoning by reframing the Winograd Schema Challenge for text-to-image models. It introduces a 500-scenario dataset generated with GPT-4 and an evaluation framework that isolates pronoun disambiguation from generic visual processing using DAAM heatmaps and IoU-based filtering. The approach reveals that even strong diffusion models like Stable Diffusion 2.0 achieve only modest pronoun-disambiguation performance (around precision) and that interpretability remains a major bottleneck, with SDXL showing notably weak heatmap reliability. The findings highlight concrete directions for improving multimodal pronoun resolution, including better entity separation, broader model diversity, bias analysis, enriched datasets, and more robust filtering techniques.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. However, applying this reasoning to multimodal domains, where understanding text and images together is essential, remains a substantial challenge. To address this, we introduce WinoVis, a novel dataset specifically designed to probe text-to-image models on pronoun disambiguation within multimodal contexts. Utilizing GPT-4 for prompt generation and Diffusion Attentive Attribution Maps (DAAM) for heatmap analysis, we propose a novel evaluation framework that isolates the models' ability in pronoun disambiguation from other visual processing challenges. Evaluation of successive model versions reveals that, despite incremental advancements, Stable Diffusion 2.0 achieves a precision of 56.7% on WinoVis, only marginally surpassing random guessing. Further error analysis identifies important areas for future research aimed at advancing text-to-image models in their ability to interpret and interact with the complex visual world.
Paper Structure (37 sections, 5 equations, 15 figures, 7 tables)

This paper contains 37 sections, 5 equations, 15 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Latent Diffusion in Image Generation
  4. Diffusion Attentive Attribution Maps
  5. Constructing WinoVis
  6. Corpus Construction Cycle
  7. Step 1: GPT Prompt Cycle
  8. Step 2: Manual Filter Process
  9. Dataset Characteristics
  10. Disparate and Distinct Entities
  11. Context Types
  12. Evaluating Pronoun Disambiguation in WinoVis
  13. Step 1: Caption Filtering
  14. Step 2: Noise Reduction in Attention Maps
  15. Step 3: Heatmap Overlap Filtering
  16. ...and 22 more sections

Figures (15)

  • Figure 1: A representative output from Stable Diffusion 2.0 on a WinoVis instance. The Diffusion Attentive Attribution Maps (DAAM) clarify the model’s focus for different terms and the correctness of its interpretation: correctly identifying 'bee' and 'flower' but erroneously associating 'it' with the bee instead of the flower.
  • Figure 2: A visual overview of the Stable Diffusion architecture, as well as the Diffusion Attention Attribution Map (DAAM) generation process.
  • Figure 3: The results of different heatmap thresholds for the prompt "The ant could not carry the leaf because it was too weak" and the term 'it'.
  • Figure 4: Illustrative example of thresholding on attention maps, progressing through stages to apply a $90^{th}$ percentile threshold, resulting in a binary mask that accentuates key attention regions.
  • Figure 5: Instances of heatmap overlap generated by Stable Diffusion 2.0 using the WinoVis dataset: On the left, two entities lead to nearly identical heatmaps, while on the right, two visually distinct entities show significant heatmap overlap.
  • ...and 10 more figures