Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are These Even Words? Quantifying the Gibberishness of Generative Speech Models

Danilo de Oliveira, Tal Peer, Jonas Rochdi, Timo Gerkmann

TL;DR

This work addresses the challenge of evaluating gibberish and phonetic implausibilities in generative speech by adopting autoregressive Speech Language Models (SpeechLMs) and using cross‑entropy and perplexity as non‑intrusive, linguistically focused metrics. By tokenizing audio into discrete semantic units and measuring the likelihood of token sequences, the authors compare several LM‑based metrics (SpeechLMScore variants, VAE‑GSLM, TWIST, SpeechGPT) and ASR+LLM pipelines against intrusion‑level baselines like WER and MOS, across clean, noisy, and gibberish speech. Key findings show that layer‑6 tokenizations improve linguistic sensitivity, that SpeechGPT performs well in noisy conditions, and that TWIST offers the most balanced overall performance, while several metrics struggle to distinguish gibberish from natural speech. The paper also releases a high‑quality gibberish speech dataset to catalyze future development of non‑intrusive measures of implausible speech, with practical implications for evaluating and curating large, in‑the‑wild speech datasets and generative systems.

Abstract

Significant research efforts are currently being dedicated to non-intrusive quality and intelligibility assessment, especially given how it enables curation of large scale datasets of in-the-wild speech data. However, with the increasing capabilities of generative models to synthesize high quality speech, new types of artifacts become relevant, such as generative hallucinations. While intrusive metrics are able to spot such sort of discrepancies from a reference signal, it is not clear how current non-intrusive methods react to high-quality phoneme confusions or, more extremely, gibberish speech. In this paper we explore how to factor in this aspect under a fully unsupervised setting by leveraging language models. Additionally, we publish a dataset of high-quality synthesized gibberish speech for further development of measures to assess implausible sentences in spoken language, alongside code for calculating scores from a variety of speech language models.

Are These Even Words? Quantifying the Gibberishness of Generative Speech Models

TL;DR

This work addresses the challenge of evaluating gibberish and phonetic implausibilities in generative speech by adopting autoregressive Speech Language Models (SpeechLMs) and using cross‑entropy and perplexity as non‑intrusive, linguistically focused metrics. By tokenizing audio into discrete semantic units and measuring the likelihood of token sequences, the authors compare several LM‑based metrics (SpeechLMScore variants, VAE‑GSLM, TWIST, SpeechGPT) and ASR+LLM pipelines against intrusion‑level baselines like WER and MOS, across clean, noisy, and gibberish speech. Key findings show that layer‑6 tokenizations improve linguistic sensitivity, that SpeechGPT performs well in noisy conditions, and that TWIST offers the most balanced overall performance, while several metrics struggle to distinguish gibberish from natural speech. The paper also releases a high‑quality gibberish speech dataset to catalyze future development of non‑intrusive measures of implausible speech, with practical implications for evaluating and curating large, in‑the‑wild speech datasets and generative systems.

Abstract

Significant research efforts are currently being dedicated to non-intrusive quality and intelligibility assessment, especially given how it enables curation of large scale datasets of in-the-wild speech data. However, with the increasing capabilities of generative models to synthesize high quality speech, new types of artifacts become relevant, such as generative hallucinations. While intrusive metrics are able to spot such sort of discrepancies from a reference signal, it is not clear how current non-intrusive methods react to high-quality phoneme confusions or, more extremely, gibberish speech. In this paper we explore how to factor in this aspect under a fully unsupervised setting by leveraging language models. Additionally, we publish a dataset of high-quality synthesized gibberish speech for further development of measures to assess implausible sentences in spoken language, alongside code for calculating scores from a variety of speech language models.
Paper Structure (10 sections, 1 equation, 2 figures, 2 tables)

This paper contains 10 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Autoregressive Speech Language Models
  4. Perplexity
  5. Metrics
  6. Analysis
  7. Data
  8. Results
  9. Conclusion
  10. References

Figures (2)

  • Figure 1: \ref{['fig:lipdiffuser_per']} DistillMOS vs. PER evaluated on LipDiffuser outputs. \ref{['fig:gibberish_utmosv2']} Normalized histogram of DistillMOS scores on gibberish speech generated by an unconditional diffusion model. Details on the data for both plots are given in \ref{['sec:data']}.
  • Figure 2: Estimated distributions of SpeechLM-based metrics on gibberish, clean and noisy (-5dB SNR) data. For UTMOSv2 and DistillMOS, higher values are better. For all others, lower is better.