Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

Hanlin Zhang, Daxin Tan, Dehua Tao, Xiao Chen, Haochen Tan, Yunhe Li, Yuchen Cao, Jianping Wang, Linqi Song

TL;DR

This work tackles the challenge of disentangling semantic content from acoustic style in discrete speech tokenization for Speech LLMs. It introduces DSA-Tokenizer, a dual-stream framework that uses ASR supervision for semantic tokens $z_s$ and mel-spectrogram reconstruction for acoustic tokens $z_a$, fused by a hierarchical Flow Matching decoder with semantic ControlNet-style and cross-attention injections. A joint reconstruction–recombination training strategy enables both high-fidelity speech reconstruction and flexible cross-utterance content–style recombination, while additional losses enforce speaker consistency and robust sampling with classifier-free guidance. The approach yields strong reconstruction quality, robust disentanglement, and improved LLM-based voice cloning, highlighting the benefits of explicit semantic–acoustic separation for controllable speech generation in fully discrete Speech LLMs with practical implications for cross-language and cross-utterance applications.

Abstract

Speech tokenizers serve as the cornerstone of discrete Speech Large Language Models (Speech LLMs). Existing tokenizers either prioritize semantic encoding, fuse semantic content with acoustic style inseparably, or achieve incomplete semantic-acoustic disentanglement. To achieve better disentanglement, we propose DSA-Tokenizer, which explicitly disentangles speech into discrete semantic and acoustic tokens via distinct optimization constraints. Specifically, semantic tokens are supervised by ASR to capture linguistic content, while acoustic tokens focus on mel-spectrograms restoration to encode style. To eliminate rigid length constraints between the two sequences, we introduce a hierarchical Flow-Matching decoder that further improve speech generation quality.Furthermore, We employ a joint reconstruction-recombination training strategy to enforce this separation. DSA-Tokenizer enables high fidelity reconstruction and flexible recombination through robust disentanglement, facilitating controllable generation in speech LLMs. Our analysis highlights disentangled tokenization as a pivotal paradigm for future speech modeling. Audio samples are avaialble at https://anonymous.4open.science/w/DSA_Tokenizer_demo/. The code and model will be made publicly available after the paper has been accepted.

DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

TL;DR

This work tackles the challenge of disentangling semantic content from acoustic style in discrete speech tokenization for Speech LLMs. It introduces DSA-Tokenizer, a dual-stream framework that uses ASR supervision for semantic tokens and mel-spectrogram reconstruction for acoustic tokens , fused by a hierarchical Flow Matching decoder with semantic ControlNet-style and cross-attention injections. A joint reconstruction–recombination training strategy enables both high-fidelity speech reconstruction and flexible cross-utterance content–style recombination, while additional losses enforce speaker consistency and robust sampling with classifier-free guidance. The approach yields strong reconstruction quality, robust disentanglement, and improved LLM-based voice cloning, highlighting the benefits of explicit semantic–acoustic separation for controllable speech generation in fully discrete Speech LLMs with practical implications for cross-language and cross-utterance applications.

Abstract

Speech tokenizers serve as the cornerstone of discrete Speech Large Language Models (Speech LLMs). Existing tokenizers either prioritize semantic encoding, fuse semantic content with acoustic style inseparably, or achieve incomplete semantic-acoustic disentanglement. To achieve better disentanglement, we propose DSA-Tokenizer, which explicitly disentangles speech into discrete semantic and acoustic tokens via distinct optimization constraints. Specifically, semantic tokens are supervised by ASR to capture linguistic content, while acoustic tokens focus on mel-spectrograms restoration to encode style. To eliminate rigid length constraints between the two sequences, we introduce a hierarchical Flow-Matching decoder that further improve speech generation quality.Furthermore, We employ a joint reconstruction-recombination training strategy to enforce this separation. DSA-Tokenizer enables high fidelity reconstruction and flexible recombination through robust disentanglement, facilitating controllable generation in speech LLMs. Our analysis highlights disentangled tokenization as a pivotal paradigm for future speech modeling. Audio samples are avaialble at https://anonymous.4open.science/w/DSA_Tokenizer_demo/. The code and model will be made publicly available after the paper has been accepted.
Paper Structure (53 sections, 12 equations, 3 figures, 4 tables)

This paper contains 53 sections, 12 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Large Language Models
  4. Disentangled Speech Tokenizers
  5. Method
  6. Dual Token Streams with Task-Specific Constraints
  7. Semantic Token Learning
  8. Acoustic Token Learning
  9. Joint Reconstruction-Recombination Training Strategy
  10. Hierarchical Flow Matching Decoder for Semantic-Acoustic Token Fusion
  11. Semantic ControlNet-Style Injection
  12. Acoustic Injection via Cross-Attention
  13. Training Objectives and Inference
  14. Flow Matching Loss
  15. Speaker Consistency Loss
  16. ...and 38 more sections

Figures (3)

  • Figure 1: Illustration of (a) speech reconstruction and (b) semantic-acoustic recombination based on discrete token
  • Figure 2: Overview of the proposed framework and training strategy. (a) DSA-Tokenizer framework: Input audio $X$ is encoded into discrete semantic and acoustic tokens, which are fed into the DiT decoder for audio generation. (b) Self-Reconstruction Mode: The model learns to predict the velocity field of the full Mel-spectrogram based on the complete acoustic and semantic tokens. (c) Recombination (Contextual Inpainting) Mode: The model learns to predict the velocity field of the masked Mel-spectrogram region based on the acoustic tokens of the unmasked region and the full semantic tokens.
  • Figure 3: Disentanglement probing evaluation results. (L0) means the first layer, (L1-7) means the second to eighth layers.