Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models

TL;DR

The paper addresses the problem that standard language-model training relies on next-token prediction and lacks interactive learning signals akin to human language acquisition. It proposes an abstract, language-only reference game in which a child-like speaker communicates with a fixed listener, receiving rewards for communicative success to study developmentally plausible learning signals. Through a feasibility study, the authors show that the reward correlates with grammaticality when the listener is a frozen QA model; they then test training from scratch and fine-tuning with two bottleneck types (length and surprisal) and observe that while interpretable shifts in speaker behavior emerge (e.g., telegraphic outputs under strong length bottlenecks), there is no improvement in linguistic evaluation metrics. The results highlight both the promise and the current limitations of interactive, cognitively motivated training for LMs, and outline concrete directions (more data, longer training, alternative architectures, multi-turn setups) for advancing this line of research toward observing interaction-driven language learning in computational cognitive models.

Abstract

We propose a method for training language models in an interactive setting inspired by child language acquisition. In our setting, a speaker attempts to communicate some information to a listener in a single-turn dialogue and receives a reward if communicative success is achieved. Unlike earlier related work using image--caption data for interactive reference games, we operationalize communicative success in a more abstract language-only question--answering setting. First, we present a feasibility study demonstrating that our reward provides an indirect signal about grammaticality. Second, we conduct experiments using reinforcement learning to fine-tune language models. We observe that cognitively plausible constraints on the communication channel lead to interpretable changes in speaker behavior. However, we do not yet see improvements on linguistic evaluations from our training regime. We outline potential modifications to the task design and training configuration that could better position future work to use our methodology to observe the benefits of interaction on language learning in computational cognitive models.

Figures (4)

• Figure 1: Illustration of the summarization game.
• Figure 2: Results from the feasibility study. $y$-axis shows macro average performance over the 17 datasets from UnifiedQA khashabi2022unifiedqav2.
• Figure 3: The trajectory of reward, penalty, and score, as well as grammatical performance during training. We plot the mean value and 95% confidence interval for all trajectories. BLiMP is a macro-average over all BLiMP categories. Grammar errors were estimated using LanguageTool. Below are textual properties of the summaries produced by the speaker throughout training.
• Figure 4: Example reward trajectories in models trained from scratch, with rolling average window of 10 steps. Runs were terminated once the model had degenerated into nonsensical output. Trajectories start after initial pretraining on next-word prediction.