Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation

Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, Xipeng Qiu

TL;DR

This paper tackles redundancies in information modeling for large-scale speech generation by proposing Chain-of-Information Generation (CoIG), which separates semantic and perceptual processing. It introduces SpeechGPT-Gen, an 8B SLLM that uses an autoregressive semantic model and a flow-matching perceptual model, with two flow-paths (Explicit and Implicit) and a semantic-prior injection to improve efficiency. Empirical results across zero-shot TTS, zero-shot voice conversion, and speech-to-speech dialogue show that the implicit-chain variant with semantic prior achieves superior WER, speaker similarity, and quality metrics, while also improving training efficiency and scalability. The approach advances efficient, scalable, and high-quality speech generation and suggests semantic-prior conditioning as a broadly useful tool for flow-based generative modeling in speech.

Abstract

Benefiting from effective speech modeling, current Speech Large Language Models (SLLMs) have demonstrated exceptional capabilities in in-context speech generation and efficient generalization to unseen speakers. However, the prevailing information modeling process is encumbered by certain redundancies, leading to inefficiencies in speech generation. We propose Chain-of-Information Generation (CoIG), a method for decoupling semantic and perceptual information in large-scale speech generation. Building on this, we develop SpeechGPT-Gen, an 8-billion-parameter SLLM efficient in semantic and perceptual information modeling. It comprises an autoregressive model based on LLM for semantic information modeling and a non-autoregressive model employing flow matching for perceptual information modeling. Additionally, we introduce the novel approach of infusing semantic information into the prior distribution to enhance the efficiency of flow matching. Extensive experimental results demonstrate that SpeechGPT-Gen markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue, underscoring CoIG's remarkable proficiency in capturing and modeling speech's semantic and perceptual dimensions. Code and models are available at https://github.com/0nutation/SpeechGPT.

SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation

TL;DR

This paper tackles redundancies in information modeling for large-scale speech generation by proposing Chain-of-Information Generation (CoIG), which separates semantic and perceptual processing. It introduces SpeechGPT-Gen, an 8B SLLM that uses an autoregressive semantic model and a flow-matching perceptual model, with two flow-paths (Explicit and Implicit) and a semantic-prior injection to improve efficiency. Empirical results across zero-shot TTS, zero-shot voice conversion, and speech-to-speech dialogue show that the implicit-chain variant with semantic prior achieves superior WER, speaker similarity, and quality metrics, while also improving training efficiency and scalability. The approach advances efficient, scalable, and high-quality speech generation and suggests semantic-prior conditioning as a broadly useful tool for flow-based generative modeling in speech.

Abstract

Benefiting from effective speech modeling, current Speech Large Language Models (SLLMs) have demonstrated exceptional capabilities in in-context speech generation and efficient generalization to unseen speakers. However, the prevailing information modeling process is encumbered by certain redundancies, leading to inefficiencies in speech generation. We propose Chain-of-Information Generation (CoIG), a method for decoupling semantic and perceptual information in large-scale speech generation. Building on this, we develop SpeechGPT-Gen, an 8-billion-parameter SLLM efficient in semantic and perceptual information modeling. It comprises an autoregressive model based on LLM for semantic information modeling and a non-autoregressive model employing flow matching for perceptual information modeling. Additionally, we introduce the novel approach of infusing semantic information into the prior distribution to enhance the efficiency of flow matching. Extensive experimental results demonstrate that SpeechGPT-Gen markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue, underscoring CoIG's remarkable proficiency in capturing and modeling speech's semantic and perceptual dimensions. Code and models are available at https://github.com/0nutation/SpeechGPT.
Paper Structure (19 sections, 7 equations, 7 figures, 4 tables)

This paper contains 19 sections, 7 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work and Background
  3. SpeechGPT-Gen
  4. Speech Tokenization
  5. LLM for Semantic Modeling
  6. Flow Matching for Perceptual Modeling
  7. Application
  8. Experiments
  9. Setups
  10. Metrics
  11. Main Results
  12. Analysis
  13. CoIG is Effective and Efficient
  14. Semantic Prior Improves Flowing Matching
  15. Scalability
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Illustration of three speech generation methods. Integrated modeling refers conductinguct semantic and perceptual modeling simultaneously. (a) Integrated Generation (b) Semantic-Disentangled Generation (c) Chain-of-Information Generation
  • Figure 2: SpeechGPT-Gen Overview. Decoder refers to SpeechTokenizer decoder. Blocks with different colors stand for representations containing different information.
  • Figure 3: Training loss of AR modeling (Left) and NAR modeling (Right) for Integrated Generation, Semantic-Disentangled Generation and Chain-of-Information Generation.
  • Figure 4: WER (Left) and speaker similarity (Right) of zero-shot TTS for Integrated Generation, Semantic-Disentangled Generation and Chain-of-Information Generation.
  • Figure 5: WER (Left) and speaker similarity (Right) of zero-shot voice conversion for flow matching with standard gaussian prior and semantic prior.
  • ...and 2 more figures