Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Acoustic BPE for Speech Generation with Discrete Tokens

Feiyu Shen, Yiwei Guo, Chenpeng Du, Xie Chen, Kai Yu

TL;DR

The paper addresses the inefficiency of modeling long sequences of discrete audio tokens derived from SSL models by introducing acoustic BPE (aBPE), which compresses token sequences and injects morphological information. A decoder-only Speech Language Model (SLM) is trained on aBPE tokens, and an acoustic rescoring method uses the SLM to select the best synthetic speech from diverse TTS outputs. Key findings show faster inference, improved syntax capture, and richer generation with aBPE, as well as effective rescore performance that aligns with human preferences, indicating practical benefits for TTS and related tasks. The approach offers a scalable, language-modeling friendly framework for speech generation using discrete tokens, with potential impact on voice cloning, speech enhancement, and other generation pipelines.

Abstract

Discrete audio tokens derived from self-supervised learning models have gained widespread usage in speech generation. However, current practice of directly utilizing audio tokens poses challenges for sequence modeling due to the length of the token sequence. Additionally, this approach places the burden on the model to establish correlations between tokens, further complicating the modeling process. To address this issue, we propose acoustic BPE which encodes frequent audio token patterns by utilizing byte-pair encoding. Acoustic BPE effectively reduces the sequence length and leverages the prior morphological information present in token sequence, which alleviates the modeling challenges of token correlation. Through comprehensive investigations on a speech language model trained with acoustic BPE, we confirm the notable advantages it offers, including faster inference and improved syntax capturing capabilities. In addition, we propose a novel rescore method to select the optimal synthetic speech among multiple candidates generated by rich-diversity TTS system. Experiments prove that rescore selection aligns closely with human preference, which highlights acoustic BPE's potential to other speech generation tasks.

Acoustic BPE for Speech Generation with Discrete Tokens

TL;DR

The paper addresses the inefficiency of modeling long sequences of discrete audio tokens derived from SSL models by introducing acoustic BPE (aBPE), which compresses token sequences and injects morphological information. A decoder-only Speech Language Model (SLM) is trained on aBPE tokens, and an acoustic rescoring method uses the SLM to select the best synthetic speech from diverse TTS outputs. Key findings show faster inference, improved syntax capture, and richer generation with aBPE, as well as effective rescore performance that aligns with human preferences, indicating practical benefits for TTS and related tasks. The approach offers a scalable, language-modeling friendly framework for speech generation using discrete tokens, with potential impact on voice cloning, speech enhancement, and other generation pipelines.

Abstract

Discrete audio tokens derived from self-supervised learning models have gained widespread usage in speech generation. However, current practice of directly utilizing audio tokens poses challenges for sequence modeling due to the length of the token sequence. Additionally, this approach places the burden on the model to establish correlations between tokens, further complicating the modeling process. To address this issue, we propose acoustic BPE which encodes frequent audio token patterns by utilizing byte-pair encoding. Acoustic BPE effectively reduces the sequence length and leverages the prior morphological information present in token sequence, which alleviates the modeling challenges of token correlation. Through comprehensive investigations on a speech language model trained with acoustic BPE, we confirm the notable advantages it offers, including faster inference and improved syntax capturing capabilities. In addition, we propose a novel rescore method to select the optimal synthetic speech among multiple candidates generated by rich-diversity TTS system. Experiments prove that rescore selection aligns closely with human preference, which highlights acoustic BPE's potential to other speech generation tasks.
Paper Structure (13 sections, 5 equations, 2 figures, 4 tables)

This paper contains 13 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Speech language model with acoustic BPE
  3. Acoustic BPE
  4. Speech language model
  5. Acoustic rescoring with SLM
  6. Experiment
  7. Experimental setup
  8. Inference speedup
  9. Syntax capturing with acoustic BPE
  10. Generation richness with acoustic BPE
  11. Text-to-speech rescoring with SLM
  12. Conclusion
  13. Acknowledgements

Figures (2)

  • Figure 1: Acoustic BPE training and encoding. (a) discretize audio into tokens leveraging the pretrained HuBERT model and $k$-means clustering. (b) convert discrete tokens into Unicode text for BPE training. (c) encode with the trained BPE model to obtain acoustic BPE tokens.
  • Figure 2: Preference test. Above: random vs. aBPE 10k; Below: w/o aBPE vs. aBPE 10k.