Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Mean Opinion Score Prediction

Hui Wang, Shiwan Zhao, Jiaming Zhou, Xiguang Zheng, Haoqin Sun, Xuechen Wang, Yong Qin

TL;DR

The paper addresses the instability of MOS prediction in open-world scenarios by introducing uncertainty modeling. It proposes an uncertainty-aware framework that combines calibrated heteroscedastic regression for aleatoric uncertainty with MC dropout for epistemic uncertainty, including a post-training calibration factor $r$ and test-time MC sampling. Empirical results on BVCC and multiple OOD benchmarks show improved uncertainty estimation, effective selective prediction, and robust OOD detection, highlighting practical benefits for real-world MOS applications. This work delivers the first MOS system with end-to-end uncertainty estimation and provides a foundation for reliable MOS prediction in diverse environments.

Abstract

Mean Opinion Score (MOS) prediction has made significant progress in specific domains. However, the unstable performance of MOS prediction models across diverse samples presents ongoing challenges in the practical application of these systems. In this paper, we point out that the absence of uncertainty modeling is a significant limitation hindering MOS prediction systems from applying to the real and open world. We analyze the sources of uncertainty in the MOS prediction task and propose to establish an uncertainty-aware MOS prediction system that models aleatory uncertainty and epistemic uncertainty by heteroscedastic regression and Monte Carlo dropout separately. The experimental results show that the system captures uncertainty well and is capable of performing selective prediction and out-of-domain detection. Such capabilities significantly enhance the practical utility of MOS systems in diverse real and open-world environments.

Uncertainty-Aware Mean Opinion Score Prediction

TL;DR

The paper addresses the instability of MOS prediction in open-world scenarios by introducing uncertainty modeling. It proposes an uncertainty-aware framework that combines calibrated heteroscedastic regression for aleatoric uncertainty with MC dropout for epistemic uncertainty, including a post-training calibration factor and test-time MC sampling. Empirical results on BVCC and multiple OOD benchmarks show improved uncertainty estimation, effective selective prediction, and robust OOD detection, highlighting practical benefits for real-world MOS applications. This work delivers the first MOS system with end-to-end uncertainty estimation and provides a foundation for reliable MOS prediction in diverse environments.

Abstract

Mean Opinion Score (MOS) prediction has made significant progress in specific domains. However, the unstable performance of MOS prediction models across diverse samples presents ongoing challenges in the practical application of these systems. In this paper, we point out that the absence of uncertainty modeling is a significant limitation hindering MOS prediction systems from applying to the real and open world. We analyze the sources of uncertainty in the MOS prediction task and propose to establish an uncertainty-aware MOS prediction system that models aleatory uncertainty and epistemic uncertainty by heteroscedastic regression and Monte Carlo dropout separately. The experimental results show that the system captures uncertainty well and is capable of performing selective prediction and out-of-domain detection. Such capabilities significantly enhance the practical utility of MOS systems in diverse real and open-world environments.
Paper Structure (15 sections, 7 equations, 5 figures, 2 tables)

This paper contains 15 sections, 7 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Uncertainty in MOS Prediction
  3. Category of Uncertainty
  4. Sources of Uncertainty
  5. Methodology
  6. Aleatoric Uncertainty Modeling
  7. Epistemic Uncertainty Modeling
  8. Experiments
  9. Experiments Setup
  10. Evaluation Metrics
  11. Experimenting with Aleatoric Uncertainty Modeling
  12. Experimenting with Epistemic Uncertainty Modeling
  13. Discussion
  14. Conclusion
  15. Acknowledgements

Figures (5)

  • Figure 1: The pipeline of our uncertainty-aware MOS prediction system, including training, calibration, and testing stages.
  • Figure 2: The error-uncertainty curve before and after calibration in BVCC test set.
  • Figure 3: The relationship between uncertainty threshold and MSE of the reliable subset in the BVCC test set.
  • Figure 4: The sharpness for epistemic distributional uncertainty and epistemic prediction uncertainty across for test sets: BVCC, BC2019 (OOD), VCC2018 (OOD), and TMHINT-QI-II (OOD).
  • Figure 5: The distribution and sharpness of epistemic distributional uncertainty on data with different noise levels.