Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation

June-Woo Kim, Dhruv Agarwal, Federica Cerina

TL;DR

The paper addresses the challenge of objective synthetic speech quality evaluation, where human Mean Opinion Score (MOS) is costly and Fréchet Distance-based metrics (FSD) depend heavily on embeddings. It systematically evaluates FSD and its kernel-based counterpart Speech Maximum Means Discrepancy (SMMD) across multiple self-supervised embeddings, noise conditions, and sample regimes, and validates them against human MOS and synthetic-trained ASR-WER via Whisper-tiny. A primary contribution is the introduction and formalization of SMMD as a complementary, nonparametric distance, alongside a rigorous embedding/dataset analysis and noise-robustness study. The findings show that WavLM Base+ yields the most stable alignment with human ratings, while FSD/SMMD serve as practical, cost-efficient proxies that complement MOS, especially when large-scale listening tests are infeasible.

Abstract

Objective evaluation of synthetic speech quality remains a critical challenge. Human listening tests are the gold standard, but costly and impractical at scale. Fréchet Distance has emerged as a promising alternative, yet its reliability depends heavily on the choice of embeddings and experimental settings. In this work, we comprehensively evaluate Fréchet Speech Distance (FSD) and its variant Speech Maximum Mean Discrepancy (SMMD) under varied embeddings and conditions. We further incorporate human listening evaluations alongside TTS intelligibility and synthetic-trained ASR WER to validate the perceptual relevance of these metrics. Our findings show that WavLM Base+ features yield the most stable alignment with human ratings. While FSD and SMMD cannot fully replace subjective evaluation, we show that they can serve as complementary, cost-efficient, and reproducible measures, particularly useful when large-scale or direct listening assessments are infeasible. Code is available at https://github.com/kaen2891/FrechetSpeechDistance.

Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation

TL;DR

The paper addresses the challenge of objective synthetic speech quality evaluation, where human Mean Opinion Score (MOS) is costly and Fréchet Distance-based metrics (FSD) depend heavily on embeddings. It systematically evaluates FSD and its kernel-based counterpart Speech Maximum Means Discrepancy (SMMD) across multiple self-supervised embeddings, noise conditions, and sample regimes, and validates them against human MOS and synthetic-trained ASR-WER via Whisper-tiny. A primary contribution is the introduction and formalization of SMMD as a complementary, nonparametric distance, alongside a rigorous embedding/dataset analysis and noise-robustness study. The findings show that WavLM Base+ yields the most stable alignment with human ratings, while FSD/SMMD serve as practical, cost-efficient proxies that complement MOS, especially when large-scale listening tests are infeasible.

Abstract

Objective evaluation of synthetic speech quality remains a critical challenge. Human listening tests are the gold standard, but costly and impractical at scale. Fréchet Distance has emerged as a promising alternative, yet its reliability depends heavily on the choice of embeddings and experimental settings. In this work, we comprehensively evaluate Fréchet Speech Distance (FSD) and its variant Speech Maximum Mean Discrepancy (SMMD) under varied embeddings and conditions. We further incorporate human listening evaluations alongside TTS intelligibility and synthetic-trained ASR WER to validate the perceptual relevance of these metrics. Our findings show that WavLM Base+ features yield the most stable alignment with human ratings. While FSD and SMMD cannot fully replace subjective evaluation, we show that they can serve as complementary, cost-efficient, and reproducible measures, particularly useful when large-scale or direct listening assessments are infeasible. Code is available at https://github.com/kaen2891/FrechetSpeechDistance.
Paper Structure (13 sections, 3 equations, 3 figures, 2 tables)

This paper contains 13 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Fréchet Speech Distance
  4. Speech embeddings and datasets for analyzing FSD
  5. Speech Maximum Mean Discrepancy distances
  6. Noise addition for quality evaluation
  7. Sample Efficiency
  8. Synthetic Speech Evaluation Experiments
  9. Settings
  10. Evaluation Metrics
  11. Comprehensive Results
  12. Correlation between Human and Objective Metrics
  13. Conclusion

Figures (3)

  • Figure 1: t-SNE results of various speech embeddings of the LS train-clean 100 hour dataset.
  • Figure 2: Comparison of the relative change for both FSD and SMMD metrics on a logarithmic scale, applied to two distinct noise sets in the LS test-clean, with SNR values ranging from 0 to 50 dB. Lower SNR values indicate lower sample quality.
  • Figure 3: Sample efficiency testing for FSD and SMMD on LS test-clean, using WavLM embeddings. Left: Behavior when subsets selected based on speakers, Right: Behavior for random subset selection.