Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DistilMOS: Layer-Wise Self-Distillation For Self-Supervised Learning Model-Based MOS Prediction

Jianing Yang, Wataru Nakata, Yuki Saito, Hiroshi Saruwatari

TL;DR

The paper tackles the generalization limits of MOS prediction models built on self-supervised speech representations, where fine-tuning can cause catastrophic forgetting. It introduces DistilMOS, a layer-wise self-distillation approach that discretizes SSL layer outputs via $k$-means and trains token predictors to classify these tokens, providing self-distillation signals in addition to MOS regression. The training objective combines MOS prediction loss with layer-wise token-classification losses, encouraging the model to leverage both acoustic and semantic information across SSL layers. Empirical results on BVCC (in-domain) and SOMOS (zero-shot) show DistilMOS consistently improves correlation-based metrics over SSL-MOS across multiple backbones, with ablations confirming the importance of token prediction and robustness to clustering choices. The method offers better generalization to unseen domains and requires no extra data during inference, highlighting practical gains for MOS prediction tasks.

Abstract

With the advancement of self-supervised learning (SSL), fine-tuning pretrained SSL models for mean opinion score (MOS) prediction has achieved state-of-the-art performance. However, during fine-tuning, these SSL-based MOS prediction models often suffer from catastrophic forgetting of the pretrained knowledge and tend to overfit the training set, resulting in poor generalization performance. In this study, we propose DistilMOS, a novel method that learns to predict not only MOS but also token IDs obtained by clustering the hidden representations of each layer in the pretrained SSL model. These layer-wise token targets serve as self-distillation signals that enables the MOS prediction model to extract rich internal knowledge from SSL models, enhancing both prediction accuracy and generalization capability. Experimental evaluations demonstrate that our method significantly outperforms standard SSL-based MOS prediction models on both in-domain and out-of-domain evaluations, verifying the effectiveness and practicality of the proposed method.

DistilMOS: Layer-Wise Self-Distillation For Self-Supervised Learning Model-Based MOS Prediction

TL;DR

The paper tackles the generalization limits of MOS prediction models built on self-supervised speech representations, where fine-tuning can cause catastrophic forgetting. It introduces DistilMOS, a layer-wise self-distillation approach that discretizes SSL layer outputs via -means and trains token predictors to classify these tokens, providing self-distillation signals in addition to MOS regression. The training objective combines MOS prediction loss with layer-wise token-classification losses, encouraging the model to leverage both acoustic and semantic information across SSL layers. Empirical results on BVCC (in-domain) and SOMOS (zero-shot) show DistilMOS consistently improves correlation-based metrics over SSL-MOS across multiple backbones, with ablations confirming the importance of token prediction and robustness to clustering choices. The method offers better generalization to unseen domains and requires no extra data during inference, highlighting practical gains for MOS prediction tasks.

Abstract

With the advancement of self-supervised learning (SSL), fine-tuning pretrained SSL models for mean opinion score (MOS) prediction has achieved state-of-the-art performance. However, during fine-tuning, these SSL-based MOS prediction models often suffer from catastrophic forgetting of the pretrained knowledge and tend to overfit the training set, resulting in poor generalization performance. In this study, we propose DistilMOS, a novel method that learns to predict not only MOS but also token IDs obtained by clustering the hidden representations of each layer in the pretrained SSL model. These layer-wise token targets serve as self-distillation signals that enables the MOS prediction model to extract rich internal knowledge from SSL models, enhancing both prediction accuracy and generalization capability. Experimental evaluations demonstrate that our method significantly outperforms standard SSL-based MOS prediction models on both in-domain and out-of-domain evaluations, verifying the effectiveness and practicality of the proposed method.
Paper Structure (13 sections, 3 equations, 3 figures, 1 table)

This paper contains 13 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. DistilMOS
  3. Overall Architecture
  4. Self-Distillation from Discretized SSL Features
  5. Training Objective
  6. Experimental evaluation
  7. Datasets
  8. Evaluation Metrics
  9. Implementation Details
  10. Baseline and Ablation Studies
  11. Results
  12. Analysis of Learned Representations
  13. Conclusion

Figures (3)

  • Figure 1: Overview of the proposed method. (a) Pretrained SSL model: frame-level representations from each encoder layer are quantized via $k$-means clustering to generate discrete token IDs. (b) Proposed MOS prediction model architecture: during training, the model is jointly optimized with two objectives, MOS regression and token prediction for self-distillation, where token predictors learn to reconstruct the quantized token IDs from each SSL layer.
  • Figure 2: Effect of the number of clusters $k$ on SRCC performance. Left: utterance-level SRCC on BVCC (in-domain) dataset. Middle: system-level SRCC on BVCC dataset. Right: utterance-level SRCC on SOMOS (zero-shot) dataset.
  • Figure 3: CCA curves between MOS prediction model representations and pretrained SSL layer outputs.