Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unseen but not Unknown: Using Dataset Concealment to Robustly Evaluate Speech Quality Estimation Models

Jaden Pieper, Stephen D. Voran

TL;DR

This work addresses the problem of robustly evaluating no-reference speech quality estimators on data outside their training distribution. It introduces Dataset Concealment (DSC), a protocol that trains Individual, Global, and Concealed models across multiple datasets and quantifies generalization through the versatility gap $v_j$ and concealment gap $c_j$, while accounting for the corpus effect. By incorporating a lightweight AlignNet Aligner, the approach mitigates label misalignment across datasets and improves performance on unseen data, as demonstrated on MOSNet, NISQA, and Wav2Vec2.0 across nine training and nine unseen datasets. The results show that Wav2Vec2.0 with an Aligner generally offers the best generalization to unseen data, and that Aligners significantly enhance inference in many cases, providing a practical framework for robust, dataset-aware speech quality estimation. The methodology offers actionable insights for model design and training data strategies in production deployments.

Abstract

We introduce Dataset Concealment (DSC), a rigorous new procedure for evaluating and interpreting objective speech quality estimation models. DSC quantifies and decomposes the performance gap between research results and real-world application requirements, while offering context and additional insights into model behavior and dataset characteristics. We also show the benefits of addressing the corpus effect by using the dataset Aligner from AlignNet when training models with multiple datasets. We demonstrate DSC and the improvements from the Aligner using nine training datasets and nine unseen datasets with three well-studied models: MOSNet, NISQA, and a Wav2Vec2.0-based model. DSC provides interpretable views of the generalization capabilities and limitations of models, while allowing all available data to be used at training. An additional result is that adding the 1000 parameter dataset Aligner to the 94 million parameter Wav2Vec model during training does significantly improve the resulting model's ability to estimate speech quality for unseen data.

Unseen but not Unknown: Using Dataset Concealment to Robustly Evaluate Speech Quality Estimation Models

TL;DR

This work addresses the problem of robustly evaluating no-reference speech quality estimators on data outside their training distribution. It introduces Dataset Concealment (DSC), a protocol that trains Individual, Global, and Concealed models across multiple datasets and quantifies generalization through the versatility gap and concealment gap , while accounting for the corpus effect. By incorporating a lightweight AlignNet Aligner, the approach mitigates label misalignment across datasets and improves performance on unseen data, as demonstrated on MOSNet, NISQA, and Wav2Vec2.0 across nine training and nine unseen datasets. The results show that Wav2Vec2.0 with an Aligner generally offers the best generalization to unseen data, and that Aligners significantly enhance inference in many cases, providing a practical framework for robust, dataset-aware speech quality estimation. The methodology offers actionable insights for model design and training data strategies in production deployments.

Abstract

We introduce Dataset Concealment (DSC), a rigorous new procedure for evaluating and interpreting objective speech quality estimation models. DSC quantifies and decomposes the performance gap between research results and real-world application requirements, while offering context and additional insights into model behavior and dataset characteristics. We also show the benefits of addressing the corpus effect by using the dataset Aligner from AlignNet when training models with multiple datasets. We demonstrate DSC and the improvements from the Aligner using nine training datasets and nine unseen datasets with three well-studied models: MOSNet, NISQA, and a Wav2Vec2.0-based model. DSC provides interpretable views of the generalization capabilities and limitations of models, while allowing all available data to be used at training. An additional result is that adding the 1000 parameter dataset Aligner to the 94 million parameter Wav2Vec model during training does significantly improve the resulting model's ability to estimate speech quality for unseen data.
Paper Structure (7 sections, 2 equations, 4 figures, 1 table)

This paper contains 7 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. The Dataset Concealment Process
  4. Datasets and Models
  5. Example Dataset Concealment Results
  6. Inference Results
  7. Conclusion

Figures (4)

  • Figure 1: Example dataset concealment results and interpretation.
  • Figure 2: DSC results for MOSNet, NISQA, and Wav2Vec models across nine datasets. Bars show performance when training conventionally and lines extending from bars show the effect of training with an Aligner. Black lines indicate statistically significant changes, grey otherwise.
  • Figure 3: DSC versatility and concealment gaps for models trained with an Aligner. Solid bars represent statistically significant gaps, otherwise bars are hatched.
  • Figure 4: Inference performance of MOSNet, NISQA, and Wav2Vec across nine unseen datasets. Bars show the performance when training conventionally. Lines extending from the bars show the effect of training with an Aligner. Black lines indicate statistically significant changes, grey otherwise.