Chronological Thinking in Full-Duplex Spoken Dialogue Language Models

TL;DR

The paper tackles the problem of inefficiency in full-duplex spoken dialogue language models arising from idle listening via repeated silence-token predictions. It introduces Chronological Thinking (CT) as a strictly causal, on-the-fly reasoning mechanism for SDLMs, implemented as ACT-R–inspired node types that replace silence with structured thinking content. The CT-SDLM architecture decouples text and audio token prediction, enabling a compact, preemptible thinking chain that amortizes reasoning during listening and incurs no extra latency. Experimental results on synthetic and real-world benchmarks show improved reasoning quality and robust turn-taking and barge-in behavior, with subjective evaluations favoring CT with thinking for both audio fidelity and response content.

Abstract

Recent advances in spoken dialogue language models (SDLMs) reflect growing interest in shifting from turn-based to full-duplex systems, where the models continuously perceive user speech streams while generating responses. This simultaneous listening and speaking design enables real-time interaction and the agent can handle dynamic conversational behaviors like user barge-in. However, during the listening phase, existing systems keep the agent idle by repeatedly predicting the silence token, which departs from human behavior: we usually engage in lightweight thinking during conversation rather than remaining absent-minded. Inspired by this, we propose Chronological Thinking, a on-the-fly conversational thinking mechanism that aims to improve response quality in full-duplex SDLMs. Specifically, chronological thinking presents a paradigm shift from conventional LLM thinking approaches, such as Chain-of-Thought, purpose-built for streaming acoustic input. (1) Strictly causal: the agent reasons incrementally while listening, updating internal hypotheses only from past audio with no lookahead. (2) No additional latency: reasoning is amortized during the listening window; once the user stops speaking, the agent halts thinking and begins speaking without further delay. Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations show consistent improvements in response quality. Furthermore, chronological thinking robustly handles conversational dynamics and attains competitive performance on full-duplex interaction metrics.

Figures (4)

• Figure 1: Comparison of a) turn-based SDLMs, b) full-duplex SDLMs c) chronological thinking in full-duplex SDLMs (ours) and d) human conversational thinking patterns.
• Figure 2: The network architecture of CT-Duplex with chronological thinking mechanism. Our model consists of a streaming speech encoder, a speech codec, an LLM backbone and a Transformer decoder. Chronological thinking happens during the listening phase.
• Figure 3: The A/B test results, including evaluations of both audio generation quality and response content quality.
• Figure 4: A dialogue example with chronological thinking chains.