Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems

Haoyuan Yu, Yuxuan Chen, Minjie Cai

TL;DR

The paper tackles natural, continuous full-duplex dialogue by reducing latency and preserving paralinguistic cues in turn-taking. It proposes a unit-based semi-cascaded architecture where a multimodal large language model directly processes audio to control within-unit listen/speak states and transitions. The framework integrates VAD, speaker verification, asynchronous ASR, and TTS in a train-free, plug-and-play pipeline, and uses a Qwen3-Omni-based decision module. On HumDial, the approach achieves state-of-the-art semantic and interaction-state inference and ranks second in Track 2, illustrating the practical viability of replacing traditional ASR-to-LLM cascades with audio-driven reasoning. Code is released publicly.

Abstract

Full-duplex voice interaction is crucial for natural human computer interaction. We present a framework that decomposes complex dialogue into minimal conversational units, enabling the system to process each unit independently and predict when to transit to the next. This framework is instantiated as a semi-cascaded full-duplex dialogue system built around a multimodal large language model, supported by auxiliary modules such as voice activity detection (VAD) and text-to-speech (TTS) synthesis. The resulting system operates in a train-free, plug-and-play manner. Experiments on the HumDial dataset demonstrate the effectiveness of our framework, which ranks second among all teams on the test set of the Human-like Spoken Dialogue Systems Challenge (Track 2: Full-Duplex Interaction). Code is available at the GitHub repository https://github.com/yu-haoyuan/fd-badcat.

Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems

TL;DR

The paper tackles natural, continuous full-duplex dialogue by reducing latency and preserving paralinguistic cues in turn-taking. It proposes a unit-based semi-cascaded architecture where a multimodal large language model directly processes audio to control within-unit listen/speak states and transitions. The framework integrates VAD, speaker verification, asynchronous ASR, and TTS in a train-free, plug-and-play pipeline, and uses a Qwen3-Omni-based decision module. On HumDial, the approach achieves state-of-the-art semantic and interaction-state inference and ranks second in Track 2, illustrating the practical viability of replacing traditional ASR-to-LLM cascades with audio-driven reasoning. Code is released publicly.

Abstract

Full-duplex voice interaction is crucial for natural human computer interaction. We present a framework that decomposes complex dialogue into minimal conversational units, enabling the system to process each unit independently and predict when to transit to the next. This framework is instantiated as a semi-cascaded full-duplex dialogue system built around a multimodal large language model, supported by auxiliary modules such as voice activity detection (VAD) and text-to-speech (TTS) synthesis. The resulting system operates in a train-free, plug-and-play manner. Experiments on the HumDial dataset demonstrate the effectiveness of our framework, which ranks second among all teams on the test set of the Human-like Spoken Dialogue Systems Challenge (Track 2: Full-Duplex Interaction). Code is available at the GitHub repository https://github.com/yu-haoyuan/fd-badcat.
Paper Structure (4 sections, 1 figure, 1 table)

This paper contains 4 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. experiments
  4. conclusion

Figures (1)

  • Figure 1: Overview of the unit-based dialogue process. Top-left shows example MLLM control behaviors; bottom-left shows intra-unit listen/speak transitions driven by continue/switch signals. Right shows one unit’s execution, where kl (keep listen) / l2s (listen-to-speak) correspond to continue/switch in the listen state and ks (keep speak) / s2l (speak-to-listen) correspond to continue/switch in the speak state.