Grounding Language in Multi-Perspective Referential Communication

TL;DR

A task and dataset for referring expression generation and comprehension in multi-agent embodied environments and the performance of automated models as speakers and listeners paired with human partners are introduced, finding that model performance in both reference generation and comprehension lags behind that of pairs of human agents.

Abstract

We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments. In this task, two agents in a shared scene must take into account one another's visual perspective, which may be different from their own, to both produce and understand references to objects in a scene and the spatial relations between them. We collect a dataset of 2,970 human-written referring expressions, each paired with human comprehension judgments, and evaluate the performance of automated models as speakers and listeners paired with human partners, finding that model performance in both reference generation and comprehension lags behind that of pairs of human agents. Finally, we experiment training an open-weight speaker model with evidence of communicative success when paired with a listener, resulting in an improvement from 58.9 to 69.3% in communicative success and even outperforming the strongest proprietary model.

Figures (6)

• Figure 1: Example scene from our environment and dataset. The center image shows the speaker on the left and the listener on the right with their respective fields of view (FOV). The speaker refers to the target object, distinguished by its blue color, and the listener selects the candidate referent they believe is described by the speaker's description, without access to its distinct color.
• Figure 2: Example scenes generated with different relative orientations ($\approx 180^\circ$ on left, $\approx 0^\circ$ on right) and with randomly- (top) or adversarially- (bottom) placed referents. Adversarially-generated referent configurations often space referents more evenly, with the target referent not easily uniquely identifiable.
• Figure 3: Analysis of referential strategies with respect to speaker agent type (top) and ranges of overlap in field of view (bottom). For each speaker agent or range of overlap, we plot the distribution over four referential strategies across all validation scenes. Within each referential strategy, we also report the proportion of generated references that guide a human listener to successfully select the target reference.
• Figure 4: Impact of task difficulty on communication errors between speaker and listener for Human, GPT, LLaVA speakers.
• Figure 5: LLaVA speaker example that leads to incorrect listener selection.