Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge

Yiwei Guo, Chenrun Wang, Yifan Yang, Hankun Wang, Ziyang Ma, Chenpeng Du, Shuai Wang, Hanzheng Li, Shuai Fan, Hui Zhang, Xie Chen, Kai Yu

TL;DR

This paper presents the SJTU X-LANCE team's participation in the Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge across TTS, SVS, and ASR tracks. It evaluates discrete token options (semantic wav2vec2.0 vs acoustic FunCodec for TTS; DAC tokens for SVS; k-means discretization on WavLM features for ASR) and designs task-specific architectures, including a VQTTS-inspired acoustic model with a phoneme-aligned vocoder, a VALL-E–based SVS pipeline, and a Zipformer RNNT-based ASR system. The TTS track achieves 1st place with both full and 1-hour data, notably using FunCodec to reach the lowest bitrate while maintaining naturalness, and demonstrates improvements when phoneme alignment is provided to the vocoder. SVS and ASR results illustrate tradeoffs between bitrate and perceptual quality and show competitive CER reductions with discretized inputs, offering practical insights into token selection, alignment strategies, and model architectures for discrete speech units. Overall, the work demonstrates that discrete speech tokens can enable high-quality speech processing at substantially reduced bitrates, with clear guidance for token choice and system design that can benefit downstream applications and research.

Abstract

Discrete speech tokens have been more and more popular in multiple speech processing fields, including automatic speech recognition (ASR), text-to-speech (TTS) and singing voice synthesis (SVS). In this paper, we describe the systems developed by the SJTU X-LANCE group for the TTS (acoustic + vocoder), SVS, and ASR tracks in the Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge. Notably, we achieved 1st rank on the leaderboard in the TTS track both with the whole training set and only 1h training data, with the highest UTMOS score and lowest bitrate among all submissions.

The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge

TL;DR

This paper presents the SJTU X-LANCE team's participation in the Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge across TTS, SVS, and ASR tracks. It evaluates discrete token options (semantic wav2vec2.0 vs acoustic FunCodec for TTS; DAC tokens for SVS; k-means discretization on WavLM features for ASR) and designs task-specific architectures, including a VQTTS-inspired acoustic model with a phoneme-aligned vocoder, a VALL-E–based SVS pipeline, and a Zipformer RNNT-based ASR system. The TTS track achieves 1st place with both full and 1-hour data, notably using FunCodec to reach the lowest bitrate while maintaining naturalness, and demonstrates improvements when phoneme alignment is provided to the vocoder. SVS and ASR results illustrate tradeoffs between bitrate and perceptual quality and show competitive CER reductions with discretized inputs, offering practical insights into token selection, alignment strategies, and model architectures for discrete speech units. Overall, the work demonstrates that discrete speech tokens can enable high-quality speech processing at substantially reduced bitrates, with clear guidance for token choice and system design that can benefit downstream applications and research.

Abstract

Discrete speech tokens have been more and more popular in multiple speech processing fields, including automatic speech recognition (ASR), text-to-speech (TTS) and singing voice synthesis (SVS). In this paper, we describe the systems developed by the SJTU X-LANCE group for the TTS (acoustic + vocoder), SVS, and ASR tracks in the Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge. Notably, we achieved 1st rank on the leaderboard in the TTS track both with the whole training set and only 1h training data, with the highest UTMOS score and lowest bitrate among all submissions.
Paper Structure (26 sections, 3 figures, 3 tables)

This paper contains 26 sections, 3 figures, 3 tables.

Table of Contents

  1. TTS Track: The VQTTS System
  2. Discrete Tokens
  3. Model Architecture
  4. Acoustic Model
  5. Discrete Unit-based Vocoder
  6. Data Preparation and Training
  7. Data Preparation
  8. Training
  9. Experimental Results
  10. SVS Track: The DOVSinger System
  11. Discrete Tokens
  12. Model Architecture
  13. Acoustic Model
  14. Discrete Unit-based Vocoder
  15. Data Preparation and Training
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Architecture of the acoustic model (left) and vocoder (right) in the TTS track. "C" in the circle means to concatenate along dimensions.
  • Figure 2: Architecture of the acoustic model (left) and vocoder (right) in the SVS track.
  • Figure 3: Illustration of the pipeline for speech discrete tokens in the ASR track.