SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Prediction?

TL;DR

SayNext-Bench introduces a cognitively grounded benchmark for next-utterance prediction from multimodal cues, revealing that current LLMs struggle to anticipate forthcoming utterances despite fluent dialogue. The framework pairs SayNext-PC datasets with a dual-route model, SayNext-Chat, that injects learnable priming tokens representing high-level priors to fuse perception with predictive processing. Across subject-dependent/independent, cross-scenario, and large-scale settings, SayNext-Chat achieves superior lexical overlap, semantic similarity, and emotion consistency relative to state-of-the-art baselines, with human evaluations endorsing its performance. This work demonstrates the feasibility and value of incorporating cognitive priors and multimodal cues to move towards more human-like, context-sensitive AI in dialogue systems.

Abstract

We explore the use of large language models (LLMs) for next-utterance prediction in human dialogue. Despite recent advances in LLMs demonstrating their ability to engage in natural conversations with users, we show that even leading models surprisingly struggle to predict a human speaker's next utterance. Instead, humans can readily anticipate forthcoming utterances based on multimodal cues, such as gestures, gaze, and emotional tone, from the context. To systematically examine whether LLMs can reproduce this ability, we propose SayNext-Bench, a benchmark that evaluates LLMs and Multimodal LLMs (MLLMs) on anticipating context-conditioned responses from multimodal cues spanning a variety of real-world scenarios. To support this benchmark, we build SayNext-PC, a novel large-scale dataset containing dialogues with rich multimodal cues. Building on this, we further develop a dual-route prediction MLLM, SayNext-Chat, that incorporates cognitively inspired design to emulate predictive processing in conversation. Experimental results demonstrate that our model outperforms state-of-the-art MLLMs in terms of lexical overlap, semantic similarity, and emotion consistency. Our results prove the feasibility of next-utterance prediction with LLMs from multimodal cues and emphasize the (i) indispensable role of multimodal cues and (ii) actively predictive processing as the foundation of natural human interaction, which is missing in current MLLMs. We hope that this exploration offers a new research entry toward more human-like, context-sensitive AI interaction for human-centered AI. Our benchmark and model can be accessed at https://saynext.github.io/.

Figures (7)

• Figure 1: Illustration of Next-Utterance Prediction in SayNext-Bench. Given a question utterance text and the corresponding human reaction video, the task requires MLLMs to predict the human’s subsequent response. Predicted responses from SayNext-Chat (green) are compared with ground-truth utterances (blue) and other MLLMs (red); key factors are extracted for interpretability. Quantitative results are reported in Sec. \ref{['sec:metrics and baseline']}.
• Figure 2: The SayNext-Chat Framework. (1) Priming factors are extracted through LLM-assisted induction to construct a priming codebook. (2) The codebook guides the LLM in assigning a target priming vector to each response. (3) During end-to-end training, the loss combines the MSE between target and predicted priming vectors with the cross-entropy loss from the LLM backbone.
• Figure 3: Comparison of SayNext-Chat with State-of-the-art Baselines. (a) Two-dimensional comparison of relative model size and performance (average rank across metrics). SayNext-Chat (red star) achieves both smaller size and higher performance. (b) Radar chart of multi-metric evaluation, where each polygon corresponds to a model and is colored consistently with (a). Our model (red) consistently ranks highest across all metrics in the radar chart. (c) Bar charts of six metrics across three dimensions. SayNext-Chat (the first bar) consistently attains the highest score.
• Figure 4: Case Study on SayNext-PC2K. In high-score samples, the predicted priming vector heatmap closely matches the target. Red and Blue indicate positive and negative values in the priming vector, with corresponding highlights in the response text. Star, heart, and drop markers denote three representative priming factors, showing their alignment between predicted and target vectors (similar colors) and clarifying their semantic meaning (listed beneath the heatmap). Low-score samples exhibit special patterns in the target priming vector that are difficult to predict.
• Figure 5: Multidimensional comparison between baseline modales and our method in subject-dependent and subject-independent protocols. Our model (red) forms the largest polygon in both subject-dependent and subject-independent settings, highlighting its outstanding performance in lexical overlap.