DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models

TL;DR

DualSpeechLM introduces an understanding-driven tokenization (USTokenizer) to align speech with text semantics and a dual-token model that uses USTokens as input and acoustic tokens as output via AcousticGPT. The framework is guided by a semantic supervision loss and a Chain of Condition to stabilize training, enabling end-to-end learning for both understanding and generation. Across eight tasks and ~4500 hours of data, the approach achieves faster convergence and superior performance over baselines, with robustness across different LLM backbones. This work offers a practical path toward unified, high-fidelity speech LLMs with strong cross-task transfer and data efficiency.

Abstract

Extending pre-trained text Large Language Models (LLMs)'s speech understanding or generation abilities by introducing various effective speech tokens has attracted great attention in the speech community. However, building a unified speech understanding and generation model still faces the following challenges: (1) Due to the huge modality gap between speech and text tokens, extending text LLMs to unified speech LLMs relies on large-scale paired data for fine-tuning, and (2) Generation and understanding tasks prefer information at different levels, e.g., generation benefits from detailed acoustic features, while understanding favors high-level semantics. This divergence leads to difficult performance optimization in one unified model. To solve these challenges, in this paper, we present two key insights in speech tokenization and speech language modeling. Specifically, we first propose an Understanding-driven Speech Tokenizer (USTokenizer), which extracts high-level semantic information essential for accomplishing understanding tasks using text LLMs. In this way, USToken enjoys better modality commonality with text, which reduces the difficulty of modality alignment in adapting text LLMs to speech LLMs. Secondly, we present DualSpeechLM, a dual-token modeling framework that concurrently models USToken as input and acoustic token as output within a unified, end-to-end framework, seamlessly integrating speech understanding and generation capabilities. Furthermore, we propose a novel semantic supervision loss and a Chain-of-Condition (CoC) strategy to stabilize model training and enhance speech generation performance. Experimental results demonstrate that our proposed approach effectively fosters a complementary relationship between understanding and generation tasks, highlighting the promising strategy of mutually enhancing both tasks in one unified model.

Figures (6)

• Figure 1: Comparison of baseline and our model on generation and understanding tasks with different ratios of generation and understanding training data.
• Figure 2: The architecture of USTokenizer, which can extract high-level semantic features aligned with text LLMs via an understanding-driven loss.
• Figure 3: DualSpeechLM’s dual-token modeling paradigm. The left illustrates the baseline pipeline treating LLM input/output as identical tokens. In contrast, our DualSpeechLM (right) incorporates an Acoustic GPT module into the text LLM module for joint training, separately processing USToken inputs and acoustic token outputs through distinct modeling paths, effectively capturing the different levels of information required for both generation and understanding tasks.
• Figure 4: Training loss curves for the understanding task. From top to bottom, the curves correspond to models trained with WavTokenizer, HuBERT, and USTokenizer as input, respectively.
• Figure 5: The details of AcousticGPT .