Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs and people both learn to form conventions -- just not with each other

Cameron R. Jones, Agnese Lombardi, Kyle Mahowald, Benjamin K. Bergen

TL;DR

The paper investigates whether large language models (LLMs) can develop conversational conventions comparable to humans in a tangram referential game. It compares Human-Human, Human-AI, and AI-AI dyads across 50 rounds to assess convention formation, demonstrating that while all dyads show gains in accuracy and lexical consistency, only same-type pairs achieve humanlike convergence, with Human-AI lagging behind and AI-AI converging through different patterns. A second experiment tests a Humanlike prompting strategy to coax more humanlike behavior from AI, finding partial improvements in length and accuracy but persistent gaps in overlap and performance relative to human-human pairs. The results imply that true conversational alignment requires more than mimicry or in-context learning; it depends on shared interpretative biases and expectations about meaning, underscoring fundamental differences in human-AI coordination and informing how to design and evaluate interactive AI systems.

Abstract

Humans align to one another in conversation -- adopting shared conventions that ease communication. We test whether LLMs form the same kinds of conventions in a multimodal communication game. Both humans and LLMs display evidence of convention-formation (increasing the accuracy and consistency of their turns while decreasing their length) when communicating in same-type dyads (humans with humans, AI with AI). However, heterogenous human-AI pairs fail -- suggesting differences in communicative tendencies. In Experiment 2, we ask whether LLMs can be induced to behave more like human conversants, by prompting them to produce superficially humanlike behavior. While the length of their messages matches that of human pairs, accuracy and lexical overlap in human-LLM pairs continues to lag behind that of both human-human and AI-AI pairs. These results suggest that conversational alignment requires more than just the ability to mimic previous interactions, but also shared interpretative biases toward the meanings that are conveyed.

LLMs and people both learn to form conventions -- just not with each other

TL;DR

The paper investigates whether large language models (LLMs) can develop conversational conventions comparable to humans in a tangram referential game. It compares Human-Human, Human-AI, and AI-AI dyads across 50 rounds to assess convention formation, demonstrating that while all dyads show gains in accuracy and lexical consistency, only same-type pairs achieve humanlike convergence, with Human-AI lagging behind and AI-AI converging through different patterns. A second experiment tests a Humanlike prompting strategy to coax more humanlike behavior from AI, finding partial improvements in length and accuracy but persistent gaps in overlap and performance relative to human-human pairs. The results imply that true conversational alignment requires more than mimicry or in-context learning; it depends on shared interpretative biases and expectations about meaning, underscoring fundamental differences in human-AI coordination and informing how to design and evaluate interactive AI systems.

Abstract

Humans align to one another in conversation -- adopting shared conventions that ease communication. We test whether LLMs form the same kinds of conventions in a multimodal communication game. Both humans and LLMs display evidence of convention-formation (increasing the accuracy and consistency of their turns while decreasing their length) when communicating in same-type dyads (humans with humans, AI with AI). However, heterogenous human-AI pairs fail -- suggesting differences in communicative tendencies. In Experiment 2, we ask whether LLMs can be induced to behave more like human conversants, by prompting them to produce superficially humanlike behavior. While the length of their messages matches that of human pairs, accuracy and lexical overlap in human-LLM pairs continues to lag behind that of both human-human and AI-AI pairs. These results suggest that conversational alignment requires more than just the ability to mimic previous interactions, but also shared interpretative biases toward the meanings that are conveyed.
Paper Structure (13 sections, 4 figures)

This paper contains 13 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Experiment 1
  3. Method
  4. Materials
  5. Participants
  6. Procedure
  7. Results
  8. Discussion
  9. Experiment 2
  10. Methods
  11. Results
  12. Discussion
  13. General Discussion

Figures (4)

  • Figure 1: The interface from the perspective of the human director (left) and the matcher (right). As directors, participants see a grid of figures and are instructed to describe the designated figure (in the green box) such that their partner, the matcher, will select it. The matcher is instructed to select the figure described by the director from the presented grid.
  • Figure 2: Examples of turns across Experiment 1 from each partnership type. AI-AI pairs (left) produce lengthy detailed geometric descriptions which do not reduce over turns. H-H pairs (right) quickly form short and analogic conventions. H-AI pairs (middle) often showed a contrast, with human partners proposing conventions that were not adopted.
  • Figure 3: Left: Accuracy increased for all partnership types in Experiment 1. AI partnerships all started below H-H pairs, but while AI-AI pairs quickly caught up, H-AI pairs did not. Center: H-H messages started shorter than other pair types and also decreased fastest. Right: Lexical overlap between descriptions of the same object from one block to the next increased for all partnership types. H-H pairs showed more overlap than H-AI pairs but less than AI-AI pairs.
  • Figure 4: Experiment 2: A new "Humanlike" prompt caused both H-AI and AI-AI pairs to produce a similar trajectory in description length to H-H pairs (center). While AI-AI pairs showed all of the hallmarks of convention-formation, H-AI pairs continued to show significantly lower accuracy (left) and lexical overlap (right) than H-H pairs.