Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Frequency-Adaptive Knowledge Distillation for Speech Enhancement

Xihao Yuan, Siqi Liu, Hanting Chen, Lu Zhou, Jian Li, Jie Hu

TL;DR

This work tackles the problem of deploying speech enhancement models on devices with limited resources by introducing Dynamic Frequency-Adaptive Knowledge Distillation (DFKD). DFKD uses a Frequency Adapter to dynamically split SE outputs into high- and low-frequency bands and applies tailored losses, including a modified cosine loss for low frequencies and a weighted high-frequency term, to guide a smaller student model. Across multiple SE architectures (DCCRN-CL, ConvTasNet, DPTNet) and datasets (DNS2020 and VoiceBank+DEMAND), DFKD yields consistent PESQ improvements over traditional logits-based KD and even matches or surpasses teacher performance in several configurations. The results demonstrate strong generalization and practical potential for edge-device SE via frequency-aware distillation.

Abstract

Deep learning-based speech enhancement (SE) models have recently outperformed traditional techniques, yet their deployment on resource-constrained devices remains challenging due to high computational and memory demands. This paper introduces a novel dynamic frequency-adaptive knowledge distillation (DFKD) approach to effectively compress SE models. Our method dynamically assesses the model's output, distinguishing between high and low-frequency components, and adapts the learning objectives to meet the unique requirements of different frequency bands, capitalizing on the SE task's inherent characteristics. To evaluate the DFKD's efficacy, we conducted experiments on three state-of-the-art models: DCCRN, ConTasNet, and DPTNet. The results demonstrate that our method not only significantly enhances the performance of the compressed model (student model) but also surpasses other logit-based knowledge distillation methods specifically for SE tasks.

Dynamic Frequency-Adaptive Knowledge Distillation for Speech Enhancement

TL;DR

This work tackles the problem of deploying speech enhancement models on devices with limited resources by introducing Dynamic Frequency-Adaptive Knowledge Distillation (DFKD). DFKD uses a Frequency Adapter to dynamically split SE outputs into high- and low-frequency bands and applies tailored losses, including a modified cosine loss for low frequencies and a weighted high-frequency term, to guide a smaller student model. Across multiple SE architectures (DCCRN-CL, ConvTasNet, DPTNet) and datasets (DNS2020 and VoiceBank+DEMAND), DFKD yields consistent PESQ improvements over traditional logits-based KD and even matches or surpasses teacher performance in several configurations. The results demonstrate strong generalization and practical potential for edge-device SE via frequency-aware distillation.

Abstract

Deep learning-based speech enhancement (SE) models have recently outperformed traditional techniques, yet their deployment on resource-constrained devices remains challenging due to high computational and memory demands. This paper introduces a novel dynamic frequency-adaptive knowledge distillation (DFKD) approach to effectively compress SE models. Our method dynamically assesses the model's output, distinguishing between high and low-frequency components, and adapts the learning objectives to meet the unique requirements of different frequency bands, capitalizing on the SE task's inherent characteristics. To evaluate the DFKD's efficacy, we conducted experiments on three state-of-the-art models: DCCRN, ConTasNet, and DPTNet. The results demonstrate that our method not only significantly enhances the performance of the compressed model (student model) but also surpasses other logit-based knowledge distillation methods specifically for SE tasks.

Paper Structure

This paper contains 10 sections, 11 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. System Overview
  4. Frequency Adapter
  5. KD Loss
  6. Experiments
  7. Datasets
  8. Implementation Details
  9. Experimental Results and Discussion
  10. Conclusions

Figures (2)

  • Figure 1: Overview of DFKD. The teacher model's data flow is represented by the red dotted line. The output is separated into high and low-frequency bands using the Frequency Adapter, and different bands are subjected to the specified loss function. $Loss_{SE}$ is a typical loss function in SE tasks, taking on different forms to accommodate the varying architectures of different networks.
  • Figure 2: Different Scenarios for Frequency Adapter. Frequency Adapter senses the characteristics of a scene and adaptively divides the frequency bands, as the distribution of high and low frequencies of human voice changes over time within one speech clip.