Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks?

Simeon Junker, Manar Ali, Larissa Koch, Sina Zarrieß, Hendrik Buschmeier

TL;DR

This work probes the pragmatic capabilities of multimodal LLMs in reference resolution tasks using abstract color stimuli. By evaluating LLaVA-NeXT, Qwen2-VL, and Janus-Pro on color patches and color grids with a director-matcher dialogue, the authors quantify how well these systems ground language in simple visual contexts. Results show that some large models approach human performance on color patches but struggle with the more complex color-grid tasks, revealing persistent gaps in contextualized color interpretation and spatial reasoning. The findings highlight architecture and prompting choices as crucial determinants of pragmatic grounding, underscoring the need for interactive grounding and targeted architectural improvements to advance situated language understanding.

Abstract

We investigate the linguistic abilities of multimodal large language models in reference resolution tasks featuring simple yet abstract visual stimuli, such as color patches and color grids. Although the task may not seem challenging for today's language models, being straightforward for human dyads, we consider it to be a highly relevant probe of the pragmatic capabilities of MLLMs. Our results and analyses indeed suggest that basic pragmatic capabilities, such as context-dependent interpretation of color descriptions, still constitute major challenges for state-of-the-art MLLMs.

Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks?

TL;DR

This work probes the pragmatic capabilities of multimodal LLMs in reference resolution tasks using abstract color stimuli. By evaluating LLaVA-NeXT, Qwen2-VL, and Janus-Pro on color patches and color grids with a director-matcher dialogue, the authors quantify how well these systems ground language in simple visual contexts. Results show that some large models approach human performance on color patches but struggle with the more complex color-grid tasks, revealing persistent gaps in contextualized color interpretation and spatial reasoning. The findings highlight architecture and prompting choices as crucial determinants of pragmatic grounding, underscoring the need for interactive grounding and targeted architectural improvements to advance situated language understanding.

Abstract

We investigate the linguistic abilities of multimodal large language models in reference resolution tasks featuring simple yet abstract visual stimuli, such as color patches and color grids. Although the task may not seem challenging for today's language models, being straightforward for human dyads, we consider it to be a highly relevant probe of the pragmatic capabilities of MLLMs. Our results and analyses indeed suggest that basic pragmatic capabilities, such as context-dependent interpretation of color descriptions, still constitute major challenges for state-of-the-art MLLMs.

Paper Structure

This paper contains 20 sections, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Experimental Setting
  4. Models
  5. Data
  6. Color Patches
  7. Color Grids
  8. Prompting Procedure
  9. Results
  10. Color Patches
  11. Color Grids
  12. Examples
  13. Discussion
  14. Conclusion
  15. Risks and Ethical Considerations
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Example color patches and color grids stimuli for director-matcher-style dyadic reference games with the human director's description and reference resolution responses of eight different MLLMs. In both examples the target referent is the object in the middle and was correctly identified by the human matcher.
  • Figure 2: Location biases in model responses for color patches (left) and color grids (right). The vertical dotted red lines denote the approximately equal distribution of target locations in the data.