Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Neural Codec Language Model for Selective Editable Text to Speech Generation

Hanchen Pei, Shujie Liu, Yanqing Liu, Jianwei Yu, Yuanhang Qian, Gongping Huang, Sheng Zhao, Yan Lu

TL;DR

SpeechEdit tackles the lack of fine-grained controllability in zero-shot TTS by reframing selectable speech editing as a prompt-guided neural codec language modeling problem. It unifies zero-shot TTS, voice conversion, and style editing within a single model by conditioning on instruction tokens and acoustic prompts, and it achieves implicit attribute disentanglement through Delta-Pair sampling trained on LibriEdit. The LibriEdit dataset, built from LibriHeavy with emotion and prosody annotations, enables scalable training for attribute-level control. Experimental results show SpeechEdit maintains naturalness and robustness while delivering state-of-the-art selective editing across multiple tasks, demonstrating the practicality of in-context learning with neural codec LMs for controllable expressive speech synthesis.

Abstract

Neural codec language models achieve impressive zero-shot Text-to-Speech (TTS) by fully imitating the acoustic characteristics of a short speech prompt, including timbre, prosody, and paralinguistic information. However, such holistic imitation limits their ability to isolate and control individual attributes. In this paper, we present a unified codec language model SpeechEdit that extends zero-shot TTS with a selective control mechanism. By default, SpeechEdit reproduces the complete acoustic profile inferred from the speech prompt, but it selectively overrides only the attributes specified by explicit control instructions. To enable controllable modeling, SpeechEdit is trained on our newly constructed LibriEdit dataset, which provides delta (difference-aware) training pairs derived from LibriHeavy. Experimental results show that our approach maintains naturalness and robustness while offering flexible and localized control over desired attributes. Audio samples are available at https://speech-editing.github.io/speech-editing/.

A Unified Neural Codec Language Model for Selective Editable Text to Speech Generation

TL;DR

SpeechEdit tackles the lack of fine-grained controllability in zero-shot TTS by reframing selectable speech editing as a prompt-guided neural codec language modeling problem. It unifies zero-shot TTS, voice conversion, and style editing within a single model by conditioning on instruction tokens and acoustic prompts, and it achieves implicit attribute disentanglement through Delta-Pair sampling trained on LibriEdit. The LibriEdit dataset, built from LibriHeavy with emotion and prosody annotations, enables scalable training for attribute-level control. Experimental results show SpeechEdit maintains naturalness and robustness while delivering state-of-the-art selective editing across multiple tasks, demonstrating the practicality of in-context learning with neural codec LMs for controllable expressive speech synthesis.

Abstract

Neural codec language models achieve impressive zero-shot Text-to-Speech (TTS) by fully imitating the acoustic characteristics of a short speech prompt, including timbre, prosody, and paralinguistic information. However, such holistic imitation limits their ability to isolate and control individual attributes. In this paper, we present a unified codec language model SpeechEdit that extends zero-shot TTS with a selective control mechanism. By default, SpeechEdit reproduces the complete acoustic profile inferred from the speech prompt, but it selectively overrides only the attributes specified by explicit control instructions. To enable controllable modeling, SpeechEdit is trained on our newly constructed LibriEdit dataset, which provides delta (difference-aware) training pairs derived from LibriHeavy. Experimental results show that our approach maintains naturalness and robustness while offering flexible and localized control over desired attributes. Audio samples are available at https://speech-editing.github.io/speech-editing/.
Paper Structure (23 sections, 3 equations, 9 figures, 3 tables)

This paper contains 23 sections, 3 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Codec LM for Speech Synthesis
  4. Controllable Speech Synthesis
  5. Proposed Method
  6. Instruction Guided Interface
  7. Data-Driven Implicit Disentanglement
  8. Instruction Composition
  9. LibriEdit Dataset
  10. Overview of LibriEdit
  11. Dataset Construction Pipeline
  12. Experiment Setup
  13. Implementation Detail
  14. Baselines and Evaluation Metrics
  15. Evaluation Results
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Overview of the SpeechEdit framework. Instruction tokens, textual content, and acoustic prompts are unified into a single token sequence through an instruction-guided conditioning interface. The codec language model performs selective attribute editing through data-driven implicit disentanglement with delta pairs.
  • Figure 2: Token sequence composition for different tasks within SpeechEdit.
  • Figure 3: Confidence thresholds for emotion labeling and the resulting distribution of emotions in LibriEdit.
  • Figure 4: Emotion editing performance on the easy task.
  • Figure 5: Average classification confidence scores for correctly predicted samples across five emotions.
  • ...and 4 more figures