LVLMs and Humans Ground Differently in Referential Communication

TL;DR

This study investigates how grounding and common ground—central to human referential communication—manifest (or fail) in multi-turn interactions between humans and LVLMs. Using a four-round director–matcher task across four partner configurations, the authors quantify communicative success, effort, and lexical entrainment, and extract referring expressions with GPT-5. The results show humans rapidly form conceptual pacts and reduce linguistic effort, while LVLMs (even GPT-5.2) fail to establish or exploit grounding across any role, including AI–AI. The work highlights significant risks for AI agents in human-facing collaboration tasks and provides a rich, openly available corpus and tooling to advance future research on grounding in vision–language models.

Abstract

For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. Here, we present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We release the online pipeline for data collection, the tools and analyses for accuracy, efficiency, and lexical overlap, and a corpus of 356 dialogues (89 pairs over 4 rounds each) that unmasks LVLMs' limitations in interactively resolving referring expressions, a crucial skill that underlies human language use.

Figures (16)

• Figure 1: Repeated referring to two baskets (non-lexicalized objects) by a human-human pair in Rounds 1-4 of our experiment, with lexical overlap highlighted in blue. Entrainment on more concise language (a conceptual pact) occurs by Round 3, after they consider multiple proposals in Rounds 1-2.
• Figure 2: Trends over four rounds for (from left to right) accuracy (%), numbers of words, number of turns, number of words referring expressions, and proportion of lexical overlap by director--matcher condition. Dots show means with 95% CIs, with each color denoting a specific pairing condition.
• Figure 3: Complete stimulus set used in the task. (a) The 12 target baskets viewed by both the director and the matcher. (b) The 6 distractor baskets viewed only by the matcher, mixed with the targets.
• Figure 4: Interface for the two-player collaborative game. The Director (left) sees the target order, while the Matcher (right) has a staging area above and the candidates below.
• Figure 5: Human-human partners explicitly acknowledging common ground, as they try to distinguish two similar baskets. The dialogue demonstrates how interlocutors use meta-linguistic cues to retrieve established conceptual pacts (e.g., "the one we worked hard on last time").