ReFESS-QI: Reference-Free Evaluation For Speech Separation With Joint Quality And Intelligibility Scoring

TL;DR

The paper tackles the problem of evaluating speech separation without references or transcripts by introducing ReFESS-QI, a text-free, reference-free estimator that predicts $SI{-}SNR$ and $WER$ from the mixture and two separated outputs using self-supervised representations. It processes three input signals through an SSL encoder, forms a $(T,3\times D_{SSL})$ representation, passes it with a Transformer, and outputs per-source and average metric estimates, trained with $MSE$ on WHAMR! using three separation systems. The approach demonstrates strong agreement with ground-truth metrics and generalizes to REAL-M, enabling fast, reference-free evaluation and the potential to guide unsupervised speech separation development. The work also shows that joint estimation of quality and intelligibility is feasible, and highlights the impact of backbone SSL models on performance, offering practical insights for scalable evaluation in real-world mixtures.

Abstract

Source separation is a crucial pre-processing step for various speech processing tasks, such as automatic speech recognition (ASR). Traditionally, the evaluation metrics for speech separation rely on the matched reference audios and corresponding transcriptions to assess audio quality and intelligibility. However, they cannot be used to evaluate real-world mixtures for which no reference exists. This paper introduces a text-free reference-free evaluation framework based on self-supervised learning (SSL) representations. The proposed framework utilize the mixture and separated tracks to predict jointly audio quality, through the Scale Invariant Signal to Noise Ratio (SI-SNR) metric, and speech intelligibility through the Word Error Rate (WER) metric. We conducted experiments on the WHAMR! dataset, which shows a WER estimation with a mean absolute error (MAE) of 17% and a Pearson correlation coefficient (PCC) of 0.77; and SI-SNR estimation with an MAE of 1.38 and PCC of 0.95. We further demonstrate the robustness of our estimator by using various SSL representations.

Figures (2)

• Figure 1: Left: Dataset Building Method. Right: Metric-estimator model's architecture, based on Semi-Supervised Learning Representations (SSLR), followed by a concatenation, a Transformer encoder layer, and a linear layer.
• Figure 2: Average WER BIN distribution (percentage) for each mixture across dataset splits