Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model as Loss: A Self-Consistent Training Paradigm

Saisamarth Rajesh Phaye, Milos Cernak, Andrew Harper

TL;DR

Problem: Conventional speech enhancement losses fail to capture subtle perceptual and task-relevant properties, limiting final output quality and generalization. Approach: Model as Loss (MAL) uses the encoder from the same encoder–decoder model as a loss, with a core objective $ \mathcal{L}_{mal} = \left\| Encoder(\mathbf{y}_{clean}) - Encoder(\mathbf{y}_{enhanced}) \right\|_1 $, and explores variants where the MAL-encoder is frozen, frozen-fe, or dynamic. Findings: MAL-based models consistently outperform pre-trained deep feature losses and hand-crafted losses on standard benchmarks, with the dynamic variant delivering the strongest NISQA gains and other variants excelling on intrusive metrics; self-consistency experiments show enhanced preservation of speech harmonics under iterative enhancement. Significance: The framework reduces reliance on large external models, improves perceptual quality and robustness to in-domain and out-of-domain noise, and is potentially transferable to domains like image denoising and medical imaging.

Abstract

Conventional methods for speech enhancement rely on handcrafted loss functions (e.g., time or frequency domain losses) or deep feature losses (e.g., using WavLM or wav2vec), which often fail to capture subtle signal properties essential for optimal performance. To address this, we propose Model as Loss, a novel training paradigm that utilizes the encoder from the same model as a loss function to guide the training. The Model as Loss paradigm leverages the encoder's task-specific feature space, optimizing the decoder to produce output consistent with perceptual and task-relevant characteristics of the clean signal. By using the encoder's learned features as a loss function, this framework enforces self-consistency between the clean reference speech and the enhanced model output. Our approach outperforms pre-trained deep feature losses on standard speech enhancement benchmarks, offering better perceptual quality and robust generalization to both in-domain and out-of-domain datasets.

Model as Loss: A Self-Consistent Training Paradigm

TL;DR

Problem: Conventional speech enhancement losses fail to capture subtle perceptual and task-relevant properties, limiting final output quality and generalization. Approach: Model as Loss (MAL) uses the encoder from the same encoder–decoder model as a loss, with a core objective , and explores variants where the MAL-encoder is frozen, frozen-fe, or dynamic. Findings: MAL-based models consistently outperform pre-trained deep feature losses and hand-crafted losses on standard benchmarks, with the dynamic variant delivering the strongest NISQA gains and other variants excelling on intrusive metrics; self-consistency experiments show enhanced preservation of speech harmonics under iterative enhancement. Significance: The framework reduces reliance on large external models, improves perceptual quality and robustness to in-domain and out-of-domain noise, and is potentially transferable to domains like image denoising and medical imaging.

Abstract

Conventional methods for speech enhancement rely on handcrafted loss functions (e.g., time or frequency domain losses) or deep feature losses (e.g., using WavLM or wav2vec), which often fail to capture subtle signal properties essential for optimal performance. To address this, we propose Model as Loss, a novel training paradigm that utilizes the encoder from the same model as a loss function to guide the training. The Model as Loss paradigm leverages the encoder's task-specific feature space, optimizing the decoder to produce output consistent with perceptual and task-relevant characteristics of the clean signal. By using the encoder's learned features as a loss function, this framework enforces self-consistency between the clean reference speech and the enhanced model output. Our approach outperforms pre-trained deep feature losses on standard speech enhancement benchmarks, offering better perceptual quality and robust generalization to both in-domain and out-of-domain datasets.

Paper Structure

This paper contains 9 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experimental setup
  4. Evaluation data
  5. Evaluation metrics
  6. Results
  7. Ablation Experiments
  8. Conclusion
  9. Acknowledgements

Figures (3)

  • Figure 1: Comparison of models using iterative enhancement. Each row represents the number of iterative enhancements applied, where the output of the previous enhancement step is used as input for the next step. The columns show different models being compared.
  • Figure 2: An illustration of the Model as Loss paradigm, showcasing the $\mathcal{L}_{\text{mal}-{\text{dynamic}}}$ variation.
  • Figure 3: NISQA MOS vs number of enhancement iterations