Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations

Xue Jiang, Xiulian Peng, Yuan Zhang, Yan Lu

TL;DR

UniCodec introduces a universal speech token learning framework that unifies linguistic and paralinguistic information into compact tokens via a low-bitrate neural codec and SSL-distilled representations. By decomposing speech into global ($m{G}$), local semantic ($m{S}$), and local residual ($m{P}$) tokens and jointly optimizing a fusion-based generative decoder, it achieves strong reconstruction quality and robust, prosody-aware generation. The approach demonstrates effectiveness across resynthesis, zero-shot TTS, S2ST, ASR, and SER on multilingual data, outperforming or matching high-bitrate baselines while significantly improving paralinguistic preservation and generation stability. This token-level framework offers a practical path toward paralinguistic-rich speech language models with compact representations suitable for cross-domain and multilingual tasks.

Abstract

Current large speech language models are mainly based on semantic tokens from discretization of self-supervised learned representations and acoustic tokens from a neural codec, following a semantic-modeling and acoustic-synthesis paradigm. However, semantic tokens discard paralinguistic attributes of speakers that is important for natural spoken communication, while prompt-based acoustic synthesis from semantic tokens has limits in recovering paralinguistic details and suffers from robustness issues, especially when there are domain gaps between the prompt and the target. This paper unifies two types of tokens and proposes the UniCodec, a universal speech token learning that encapsulates all semantics of speech, including linguistic and paralinguistic information, into a compact and semantically-disentangled unified token. Such a unified token can not only benefit speech language models in understanding with paralinguistic hints but also help speech generation with high-quality output. A low-bitrate neural codec is leveraged to learn such disentangled discrete representations at global and local scales, with knowledge distilled from self-supervised learned features. Extensive evaluations on multilingual datasets demonstrate its effectiveness in generating natural, expressive and long-term consistent output quality with paralinguistic attributes well preserved in several speech processing tasks.

Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations

TL;DR

UniCodec introduces a universal speech token learning framework that unifies linguistic and paralinguistic information into compact tokens via a low-bitrate neural codec and SSL-distilled representations. By decomposing speech into global (), local semantic (), and local residual () tokens and jointly optimizing a fusion-based generative decoder, it achieves strong reconstruction quality and robust, prosody-aware generation. The approach demonstrates effectiveness across resynthesis, zero-shot TTS, S2ST, ASR, and SER on multilingual data, outperforming or matching high-bitrate baselines while significantly improving paralinguistic preservation and generation stability. This token-level framework offers a practical path toward paralinguistic-rich speech language models with compact representations suitable for cross-domain and multilingual tasks.

Abstract

Current large speech language models are mainly based on semantic tokens from discretization of self-supervised learned representations and acoustic tokens from a neural codec, following a semantic-modeling and acoustic-synthesis paradigm. However, semantic tokens discard paralinguistic attributes of speakers that is important for natural spoken communication, while prompt-based acoustic synthesis from semantic tokens has limits in recovering paralinguistic details and suffers from robustness issues, especially when there are domain gaps between the prompt and the target. This paper unifies two types of tokens and proposes the UniCodec, a universal speech token learning that encapsulates all semantics of speech, including linguistic and paralinguistic information, into a compact and semantically-disentangled unified token. Such a unified token can not only benefit speech language models in understanding with paralinguistic hints but also help speech generation with high-quality output. A low-bitrate neural codec is leveraged to learn such disentangled discrete representations at global and local scales, with knowledge distilled from self-supervised learned features. Extensive evaluations on multilingual datasets demonstrate its effectiveness in generating natural, expressive and long-term consistent output quality with paralinguistic attributes well preserved in several speech processing tasks.

Paper Structure

This paper contains 26 sections, 16 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Representation Learning
  4. Generative Speech Language Modeling
  5. Neural Speech Codecs
  6. Paralinguistics-Enhanced Speech Processing
  7. The Proposed UniCodec
  8. Problem Formulation
  9. UniCodec
  10. Encoder
  11. Vector Quantization with Rate Control
  12. Decoder
  13. Global and Local Fusion
  14. Generative Decoder
  15. Acoustic Decoder
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Illustration of different types of discrete speech representations and their typical usage in generative speech language modeling. Our unified token encapsulates disentangled semantics into a compact token. LM: language modeling.
  • Figure 2: (a) The proposed framework. (b) Local disentangled encoders and local fusion.
  • Figure 3: Subjective evaluation results. The red dotted line represents the score of the reference. The error bar denotes $95\%$ confidence intervals.