APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech

TL;DR

This paper tackles automatic MOS prediction for synthetic speech by addressing gaps in aligning predictions with human perception due to neglect of auditory processing and semantic degradation. It introduces APG-MOS, which integrates a cochlea-inspired auditory module to encode signals into electrochemical representations, a RVQ-based semantic distortion space, and a residual cross-attention fusion with a progressive learning strategy. Experiments on BVCC and SOMOS show superior system-level agreement with human judgments and favorable model efficiency compared with strong baselines. The work advances a biologically plausible SQA framework that benefits from combining expert auditory knowledge with SSL-based semantic modeling, offering robust MOS predictions across diverse synthetic speech systems.

Abstract

Automatic speech quality assessment aims to quantify subjective human perception of speech through computational models to reduce the need for labor-consuming manual evaluations. While models based on deep learning have achieved progress in predicting mean opinion scores (MOS) to assess synthetic speech, the neglect of fundamental auditory perception mechanisms limits consistency with human judgments. To address this issue, we propose an auditory perception guided-MOS prediction model (APG-MOS) that synergistically integrates auditory modeling with semantic analysis to enhance consistency with human judgments. Specifically, we first design a perceptual module, grounded in biological auditory mechanisms, to simulate cochlear functions, which encodes acoustic signals into biologically aligned electrochemical representations. Secondly, we propose a residual vector quantization (RVQ)-based semantic distortion modeling method to quantify the degradation of speech quality at the semantic level. Finally, we design a residual cross-attention architecture, coupled with a progressive learning strategy, to enable multimodal fusion of encoded electrochemical signals and semantic representations. Experiments demonstrate that APG-MOS achieves superior performance on two primary benchmarks. Our code and checkpoint will be available on a public repository upon publication.

Figures (5)

• Figure 1: Illustration of the SQA Task. MOS is a widely adopted metric to assess speech quality for speech synthesis systems such as TTS and VC systems. The subjective listening test for obtaining MOS is a labor-intensive process where each synthetic speech from each system is scored by multiple listeners, whereas automatic SQA can efficiently predict MOS through a computational model. Our proposed APG-MOS is an automatic SQA model which aims to approximate human perception by jointly leveraging auditory perception modeling and semantic modeling.
• Figure 2: The main architecture of the proposed APG-MOS method. In the frst stage, auditory perception modeling is proposed to simulate cochlear mechano-electric transduction to encode speech signals into electrochemical representations (Section \ref{['sec:auditory']}). In the second stage, semantic modeling method is proposed by leveraging HuBERT-RVQ and fine-tuning Wav2vec2 model (Section \ref{['sec:semantic']}). In the final stage, joint cognitive modeling is proposed to integrate auditory and semantic representations via a residual cross-attention architecture (Section \ref{['sec:cognitive']}).
• Figure 3: Illustration of the attention mask.
• Figure 4: Illustration of the progressive training strategy.
• Figure 5: Visualization of the Mel spectrogram, cochleagram $X_{\text{ele}}$, weight heatmap of cross-attention blocks for the first and the last layer of a sample "sys520c6-utt3f40b05.wav" in BVCC dataset. The part encased in the dashed box can illustrate the impact of auditory features in guiding the weights of SSL model-based features from shallow to deep.