Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contract-Driven QoE Auditing for Speech and Singing Services: From MOS Regression to Service Graphs

Wenzhang Du

TL;DR

The paper addresses the limitations of MOS as a sole QoE metric by introducing contract-driven QoE auditing over deployment service graphs. It defines experience contracts as Boolean predicates on rating vectors and computes a contract satisfaction vector Q(G, C) to enable interpretable, graph-aware auditing that remains stable under view changes. Through experiments on URGENT2024 MOS and SingMOS, it shows MOS regression is a special case, contract-driven metrics offer better view stability, and learning efficiency is driven by semantic alignment between contracts and evidence. The framework enables deployment-aligned QoE evaluation with richer explanations beyond scalar MOS, supported by self-supervised audio representations and rating metadata analyses.

Abstract

Subjective mean opinion scores (MOS) remain the de-facto target for non-intrusive speech and singing quality assessment. However, MOS is a scalar that collapses heterogeneous user expectations, ignores service-level objectives, and is difficult to compare across deployment graphs. We propose a contract-driven QoE auditing framework: each service graph G is evaluated under a set of human-interpretable experience contracts C, yielding a contract-level satisfaction vector Q(G, C). We show that (i) classical MOS regression is a special case with a degenerate contract set, (ii) contract-driven quality is more stable than MOS under graph view transformations (e.g., pooling by system vs. by system type), and (iii) the effective sample complexity of learning contracts is governed by contract semantics rather than merely the dimensionality of C. We instantiate the framework on URGENT2024 MOS (6.9k speech utterances with raw rating vectors) and SingMOS v1 (7,981 singing clips; 80 systems). On URGENT, we train a contract-aware neural auditor on self-supervised WavLM embeddings; on SingMOS, we perform contract-driven graph auditing using released rating vectors and metadata without decoding audio. Empirically, our auditor matches strong MOS predictors in MOS accuracy while providing calibrated contract probabilities; on SingMOS, Q(G, C) exhibits substantially smaller cross-view drift than raw MOS and graph-only baselines; on URGENT, difficulty curves reveal that mis-specified "simple" contracts can be harder to learn than richer but better aligned contract sets.

Contract-Driven QoE Auditing for Speech and Singing Services: From MOS Regression to Service Graphs

TL;DR

The paper addresses the limitations of MOS as a sole QoE metric by introducing contract-driven QoE auditing over deployment service graphs. It defines experience contracts as Boolean predicates on rating vectors and computes a contract satisfaction vector Q(G, C) to enable interpretable, graph-aware auditing that remains stable under view changes. Through experiments on URGENT2024 MOS and SingMOS, it shows MOS regression is a special case, contract-driven metrics offer better view stability, and learning efficiency is driven by semantic alignment between contracts and evidence. The framework enables deployment-aligned QoE evaluation with richer explanations beyond scalar MOS, supported by self-supervised audio representations and rating metadata analyses.

Abstract

Subjective mean opinion scores (MOS) remain the de-facto target for non-intrusive speech and singing quality assessment. However, MOS is a scalar that collapses heterogeneous user expectations, ignores service-level objectives, and is difficult to compare across deployment graphs. We propose a contract-driven QoE auditing framework: each service graph G is evaluated under a set of human-interpretable experience contracts C, yielding a contract-level satisfaction vector Q(G, C). We show that (i) classical MOS regression is a special case with a degenerate contract set, (ii) contract-driven quality is more stable than MOS under graph view transformations (e.g., pooling by system vs. by system type), and (iii) the effective sample complexity of learning contracts is governed by contract semantics rather than merely the dimensionality of C. We instantiate the framework on URGENT2024 MOS (6.9k speech utterances with raw rating vectors) and SingMOS v1 (7,981 singing clips; 80 systems). On URGENT, we train a contract-aware neural auditor on self-supervised WavLM embeddings; on SingMOS, we perform contract-driven graph auditing using released rating vectors and metadata without decoding audio. Empirically, our auditor matches strong MOS predictors in MOS accuracy while providing calibrated contract probabilities; on SingMOS, Q(G, C) exhibits substantially smaller cross-view drift than raw MOS and graph-only baselines; on URGENT, difficulty curves reveal that mis-specified "simple" contracts can be harder to learn than richer but better aligned contract sets.

Paper Structure

This paper contains 23 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions (Hypotheses P0--P2)
  3. Related Work
  4. MOS collection and standardization
  5. Non-intrusive MOS prediction
  6. Contracts, fairness, and graph views
  7. Contract-Driven QoE Auditing
  8. Service graph and rating model
  9. Experience contracts and contract vectors
  10. Graph views and view stability
  11. MOS regression as a special case (P0)
  12. Experimental Setup
  13. Datasets
  14. Feature extraction
  15. Models and baselines (URGENT)
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Explanatory gain on URGENT: within MOS bins, contract satisfaction $Q_{\text{total}}$ can vary widely, revealing heterogeneity invisible to scalar MOS.
  • Figure 2: SingMOS view stability: mean absolute system--type drift for MOS, a graph-only MOS baseline, and contract metrics (with 95% bootstrap intervals). Contract-driven metrics drift less.
  • Figure 3: Contract learning difficulty on URGENT vs. training fraction. Difficulty is governed by semantic alignment, not only $|\mathcal{C}|$.