A Full-duplex Speech Dialogue Scheme Based On Large Language Models

TL;DR

The paper introduces a full-duplex dialogue framework built around a single LLM that manages a neural finite-state machine with SPEAK and LISTEN states, guided by perception (ASR) and motor (TTS) modules to enable simultaneous speaking and listening. It combines instruction tuning and prompt engineering to train the LLM to autonomously control dialogue flow via control tokens, achieving dramatically lower latency than traditional half-duplex systems. A new duplex-dialogue-3k benchmark and a streaming-enabled simulator provide automated assessments of interruption timing and quality, showing competitive interruption precision and robust responsiveness for an 8B parameter model. The work highlights the potential of simplifying end-to-end speech interaction by moving toward multimodal LLMs capable of integrating perception and speech synthesis more tightly, while acknowledging current reliance on ASR/TTS latency.

Abstract

We present a generative dialogue system capable of operating in a full-duplex manner, allowing for seamless interaction. It is based on a large language model (LLM) carefully aligned to be aware of a perception module, a motor function module, and the concept of a simple finite state machine (called neural FSM) with two states. The perception and motor function modules operate in tandem, allowing the system to speak and listen to the user simultaneously. The LLM generates textual tokens for inquiry responses and makes autonomous decisions to start responding to, wait for, or interrupt the user by emitting control tokens to the neural FSM. All these tasks of the LLM are carried out as next token prediction on a serialized view of the dialogue in real-time. In automatic quality evaluations simulating real-life interaction, the proposed system reduces the average conversation response latency by more than threefold compared with LLM-based half-duplex dialogue systems while responding within less than 500 milliseconds in more than 50% of evaluated interactions. Running an LLM with only 8 billion parameters, our system exhibits an 8% higher interruption precision rate than the best available commercial LLM for voice-based dialogue.

Figures (2)

• Figure 1: Left. Overview of the agent design that enables LLM-based full-duplex dialogue models. The agent is equipped with one LLM, one perception module, and one motor function module. The latter operates continuously and simultaneously to collect input to the LLM and produce voice-based LLM outputs. Right. The LLM operates a two-state neural FSM with $\mathbf{SPEAK}$ and $\mathbf{LISTEN}$ states. At each timestep, the LLM either 1) receives an external input token, 2) generates a textual token for speech, or 3) produces a control token to signal state transition in the neural FSM. This simple workflow enables full-duplex dialogue without any external moderation module.
• Figure 2: In LLM based full-duplex dialogue system, the LLM operates a two state FSM, governing state transitions in the dialogue.