Learning to Simulate Human Dialogue

TL;DR

This work treats next-turn dialogue prediction as a proxy for simulating human behavior, examining how thinking and reward design affect alignment with real conversations. It contrasts two learning signals—LLM-as-a-judge and direct log-probability optimization—and two reasoning modes (with or without chain-of-thought), deriving an ELBO-based objective for the latent-CoT setting. The key finding is that judge-based rewards tend to cause reward hacking and poor ground-truth alignment, especially when thinking is allowed, while directly maximizing the ground-truth likelihood, particularly with chain-of-thought treated as a latent variable, yields stronger predictions of human dialogue and higher human-preference win rates. This suggests that distribution-matching objectives grounded in real human data, possibly scaled to more diverse corpora, offer a principled path to modeling nuanced human behavior in conversation.

Abstract

To predict what someone will say is to model how they think. We study this through next-turn dialogue prediction: given a conversation, predict the next utterance produced by a person. We compare learning approaches along two dimensions: (1) whether the model is allowed to think before responding, and (2) how learning is rewarded either through an LLM-as-a-judge that scores semantic similarity and information completeness relative to the ground-truth response, or by directly maximizing the log-probability of the true human dialogue. We find that optimizing for judge-based rewards indeed increases judge scores throughout training, however it decreases the likelihood assigned to ground truth human responses and decreases the win rate when human judges choose the most human-like response among a real and synthetic option. This failure is amplified when the model is allowed to think before answering. In contrast, by directly maximizing the log-probability of observed human responses, the model learns to better predict what people actually say, improving on both log-probability and win rate evaluations. Treating chain-of-thought as a latent variable, we derive a lower bound on the log-probability. Optimizing this objective yields the best results on all our evaluations. These results suggest that thinking helps primarily when trained with a distribution-matching objective grounded in real human dialogue, and that scaling this approach to broader conversational data may produce models with a more nuanced understanding of human behavior.

Figures (7)

• Figure 1: (left)Learning to simulate human dialogue. Predicting the next dialogue $y$ given context $x$, optionally reasoning via chain-of-thought $z$. (center) We vary the reward signal (LLM-as-a-judge vs. log-probability) and reasoning mode (thinking vs. no thinking). With thinking, chain-of-thought serves as a latent variable optimized to increase likelihood of true human responses. (right) Log-probability training yields higher ground-truth likelihood on the test set and human win rates; judge-based training leads to reward hacking. Thinking amplifies both effects.
• Figure 2: Reward hacking with LLM-as-a-judge. Judge rewards increase throughout training for both thinking and no-thinking methods, with thinking achieving higher final rewards. However, this improvement is due to models learning to exploit the judge rather than becoming better at predicting human dialogue.
• Figure 3: Training to optimize log-probability of true responses. Log-probability of the true human dialogue improves quickly in the first 100 steps, then stabilizes. Training with CoTs is better than directly finetuning on the true human dialogues, showing that thinking helps when optimized with a distribution matching objective.
• Figure 4: Qualitative Comparison of Models.(left) A mother-child conversation about illness. The RL model captures maternal intent by staying home with the sick child; SFT suggests the child go to the doctor alone. (right) A conversation about a lost pet. The RL model responds with empathy to a lost pet; SFT asks a question already answered.
• Figure 5: Examples of reward hacking behavior in LLM-as-a-Judge trained models.