UrgentMOS: Unified Multi-Metric and Preference Learning for Robust Speech Quality Assessment

TL;DR

UrgentMOS tackles the fragility of MOS-based speech quality evaluation in real-world, multi-source settings by jointly learning from multiple objective and perceptual metrics while tolerating incomplete annotations. It introduces a unified framework with an Absolute Metric Prediction Module and a Naturalness-Conditioned Preference Module that share a multi-encoder feature extractor and support both absolute scores and pairwise preferences, including CMOS, through a range-constrained and cross-attentive design. The paper derives preference supervision from abundant ACR data and complements it with symmetric pair construction to stabilize learning, plus releases a preference-derived dataset to benchmark comparative judgments. Empirical results across diverse datasets (TTS, SE, distortions) show state-of-the-art performance in both absolute and comparative evaluation, demonstrating improved robustness and practical benchmarking capabilities for modern speech systems.

Abstract

Automatic speech quality assessment has become increasingly important as modern speech generation systems continue to advance, while human listening tests remain costly, time-consuming, and difficult to scale. Most existing learning-based assessment models rely primarily on scarce human-annotated mean opinion score (MOS) data, which limits robustness and generalization, especially when training across heterogeneous datasets. In this work, we propose UrgentMOS, a unified speech quality assessment framework that jointly learns from diverse objective and perceptual quality metrics, while explicitly tolerating the absence of arbitrary subsets of metrics during training. By leveraging complementary quality facets under heterogeneous supervision, UrgentMOS enables effective utilization of partially annotated data and improves robustness when trained on large-scale, multi-source datasets. Beyond absolute score prediction, UrgentMOS explicitly models pairwise quality preferences by directly predicting comparative MOS (CMOS), making it well suited for preference-based evaluation scenarios commonly adopted in system benchmarking. Extensive experiments across a wide range of speech quality datasets, including simulated distortions, speech enhancement, and speech synthesis, demonstrate that UrgentMOS consistently achieves state-of-the-art performance in both absolute and comparative evaluation settings.

Figures (5)

• Figure 1: Overview of the UrgentMOS architecture and training paradigm. UrgentMOS consists of an Absolute Metric Prediction Module (AMPM) and a Naturalness-Conditioned Preference Module (NCPM). Each speech sample is processed by a shared feature extractor; the two AMPM blocks shown for Audio A and Audio B correspond to the same AMPM with shared parameters. AMPM predicts multiple objective and perceptual quality metrics via shared metric encoders and metric-specific heads, while NCPM operates on representations from the naturalness-related metric group to model pairwise quality preferences using cross-attention. $\mathcal{L}_{CE}$ denotes cross entropy loss. $\mathcal{L}_{MSE}$ is detailed in (\ref{['eq:loss_mse']}).
• Figure 2: Feature extractor in UrgentMOS. Representations are temporally aligned via interpolation before fusion.
• Figure 3: Spearman correlation across different speech quality metrics on the Urgent2025-SQA dataset.
• Figure 4: Preference accuracy on the SpeechEval test set versus tie threshold $\delta$. Direct preference models are threshold-invariant, while MOS predictors are highly sensitive.
• Figure 5: Preference accuracy on the SpeechJudge test set versus tie threshold $\delta$.