Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model

Siyang Wang, Éva Székely

TL;DR

This paper evaluates Bark, a large discrete-token speech language model for text-to-speech synthesis, across five dimensions: speaking style, intelligibility, speaker consistency, prosodic variation, and spontaneous behaviour. It compares Bark to the multi-speaker VITS baseline using automatic metrics and listening tests on LibriTTS read speech and DailyDialog conversational text. Bark demonstrates highly varied and natural prosody and spontaneous output, often outperforming conventional TTS in naturalness and contextual suitability, but exhibits weaker intelligibility and speaker consistency, with robustness improving modestly as the model scales. The work provides a rigorous benchmark framework for future generative SLMs in speech synthesis and discusses implications for multilingual capabilities and scaling strategies.

Abstract

Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. These speech language models (SLMs), similarly to their textual counterparts, are scalable, probabilistic, and context-aware. While they can produce diverse and natural outputs, they sometimes face issues such as unintelligibility and the inclusion of non-speech noises or hallucination. As the adoption of this innovative paradigm in speech synthesis increases, there is a clear need for an in-depth evaluation of its capabilities and limitations. In this paper, we evaluate TTS from a discrete token-based SLM, through both automatic metrics and listening tests. We examine five key dimensions: speaking style, intelligibility, speaker consistency, prosodic variation, spontaneous behaviour. Our results highlight the model's strength in generating varied prosody and spontaneous outputs. It is also rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. However, the model's performance in intelligibility and speaker consistency lags behind traditional TTS. Additionally, we show that increasing the scale of SLMs offers a modest boost in robustness. Our findings aim to serve as a benchmark for future advancements in generative SLMs for speech synthesis.

Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model

TL;DR

This paper evaluates Bark, a large discrete-token speech language model for text-to-speech synthesis, across five dimensions: speaking style, intelligibility, speaker consistency, prosodic variation, and spontaneous behaviour. It compares Bark to the multi-speaker VITS baseline using automatic metrics and listening tests on LibriTTS read speech and DailyDialog conversational text. Bark demonstrates highly varied and natural prosody and spontaneous output, often outperforming conventional TTS in naturalness and contextual suitability, but exhibits weaker intelligibility and speaker consistency, with robustness improving modestly as the model scales. The work provides a rigorous benchmark framework for future generative SLMs in speech synthesis and discusses implications for multilingual capabilities and scaling strategies.

Abstract

Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. These speech language models (SLMs), similarly to their textual counterparts, are scalable, probabilistic, and context-aware. While they can produce diverse and natural outputs, they sometimes face issues such as unintelligibility and the inclusion of non-speech noises or hallucination. As the adoption of this innovative paradigm in speech synthesis increases, there is a clear need for an in-depth evaluation of its capabilities and limitations. In this paper, we evaluate TTS from a discrete token-based SLM, through both automatic metrics and listening tests. We examine five key dimensions: speaking style, intelligibility, speaker consistency, prosodic variation, spontaneous behaviour. Our results highlight the model's strength in generating varied prosody and spontaneous outputs. It is also rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. However, the model's performance in intelligibility and speaker consistency lags behind traditional TTS. Additionally, we show that increasing the scale of SLMs offers a modest boost in robustness. Our findings aim to serve as a benchmark for future advancements in generative SLMs for speech synthesis.
Paper Structure (28 sections, 4 figures, 3 tables)

This paper contains 28 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Evaluated Model: Bark
  3. Model Architecture
  4. Baseline: VITS
  5. Parameter Count and Runtime
  6. Evaluation Method
  7. Text Input: Two Speaking Styles
  8. Read-Speech Text: LibriTTS
  9. Conversational Text: DailyDialog
  10. Intelligibility
  11. Speaker Consistency
  12. Prosodic Variation
  13. Spontaneous Behaviour
  14. Listening Tests
  15. Test 1: MOS-N(aturalness)
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Illustration of Bark.
  • Figure 2: Word error rate (WER). Speaker denotation is b[x] for Bark speakers, p[x] for VITS speakers. Speaker order is based on ascending-sorted WER.
  • Figure 3: Speaker similarity metrics. VITS speakers are randomly selected to match the number of speakers in Bark for clear comparison. Color corresponds to the mean similarity calculated by a speaker embedding model ECAPA-TDNN desplanques2020ecapa.
  • Figure 4: Prosodic variation.