Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

Manar Ali, Judith Sieker, Sina Zarrieß, Hendrik Buschmeier

TL;DR

This work investigates whether vision-language models can express their internal uncertainty through explicit clarification in a controlled setting. By employing color-grid reference games as a testbed, the authors compare baseline addressee behavior with an explicit clarification-enabled protocol across three models, including GPT-5-mini and Qwen-2.5-VL variants. Findings show limited alignment between uncertainty and clarifying behavior, with GPT-5-mini showing some calibration while Qwen models largely fail to translate uncertainty into useful clarification. The study demonstrates the value of reference games for probing pragmatic and interactive capabilities of vision-language models and highlights a direction for improving uncertainty signaling and grounding in AI dialogue systems.

Abstract

In human conversation, both interlocutors play an active role in maintaining mutual understanding. When addressees are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar addressee role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a good testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.

Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

TL;DR

This work investigates whether vision-language models can express their internal uncertainty through explicit clarification in a controlled setting. By employing color-grid reference games as a testbed, the authors compare baseline addressee behavior with an explicit clarification-enabled protocol across three models, including GPT-5-mini and Qwen-2.5-VL variants. Findings show limited alignment between uncertainty and clarifying behavior, with GPT-5-mini showing some calibration while Qwen models largely fail to translate uncertainty into useful clarification. The study demonstrates the value of reference games for probing pragmatic and interactive capabilities of vision-language models and highlights a direction for improving uncertainty signaling and grounding in AI dialogue systems.

Abstract

In human conversation, both interlocutors play an active role in maintaining mutual understanding. When addressees are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar addressee role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a good testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.
Paper Structure (22 sections, 4 figures, 4 tables)

This paper contains 22 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Models and Data
  5. Experiments
  6. Human Data
  7. Baseline Experiment
  8. Clarification Experiment
  9. Results
  10. Discussion and Conclusion
  11. Implementation Details
  12. Scientific Artifacts
  13. Prompts
  14. Baseline Experiment Prompt
  15. Clarification Experiment Prompt
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Example item from the dataset. The speaker referred to the second item with the description “Bottom left is bright pink.”
  • Figure 2: Sankey diagrams for GPT-5-mini (left) and Qwen2.5 VL-72B (right) showing each model’s outcomes in the baseline and clarification experiments. The flow indicates for which baseline items clarification requests were generated and how consistent responses were.
  • Figure 3: Example of GPT-5-mini behavior in the clarification experiment.
  • Figure 4: Example of Qwen2.5-VL-72B behavior with human-in-the-loop interaction.