Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement

Runduo Han, Weiming Xu, Zihan Zhang, Mingshuai Liu, Lei Xie

TL;DR

This work tackles the challenge of deploying high‑performing speech enhancement in resource‑constrained scenarios by distilling a large DCCRN‑family model into a small yet powerful student. The authors introduce Distil‑DCCRN, a lightweight TF‑domain SE model trained via AT‑KL knowledge distillation from a stronger Uformer teacher, transferring both outputs and misaligned intermediate features across time and channel dimensions. The method combines time and channel dimension attention transfer with KL divergence and a balanced SI‑SNR loss, enabling effective cross‑model knowledge transfer despite differing STFT configurations. On the DNS dataset, Distil‑DCCRN achieves DNSMOS comparable to DCCRN while using only 1.1M parameters (about 30% of DCCRN) and delivering improved PESQ and SI‑SNR, with FLOPs roughly halved, demonstrating the practicality and effectiveness of AT‑KL KD for lightweight speech enhancement. Overall, the approach enables small, simple architectures to rival state‑of‑the‑art lightweight models, extending the reach of strong SE performance to parameter‑limited applications.

Abstract

The deep complex convolution recurrent network (DCCRN) achieves excellent speech enhancement performance by utilizing the audio spectrum's complex features. However, it has a large number of model parameters. We propose a smaller model, Distil-DCCRN, which has only 30% of the parameters compared to the DCCRN. To ensure that the performance of Distil-DCCRN matches that of the DCCRN, we employ the knowledge distillation (KD) method to use a larger teacher model to help train a smaller student model. We design a knowledge distillation (KD) method, integrating attention transfer and Kullback-Leibler divergence (AT-KL) to train the student model Distil-DCCRN. Additionally, we use a model with better performance and a more complicated structure, Uformer, as the teacher model. Unlike previous KD approaches that mainly focus on model outputs, our method also leverages the intermediate features from the models' middle layers, facilitating rich knowledge transfer across different structured models despite variations in layer configurations and discrepancies in the channel and time dimensions of intermediate features. Employing our AT-KL approach, Distil-DCCRN outperforms DCCRN as well as several other competitive models in both PESQ and SI-SNR metrics on the DNS test set and achieves comparable results to DCCRN in DNSMOS.

Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement

TL;DR

This work tackles the challenge of deploying high‑performing speech enhancement in resource‑constrained scenarios by distilling a large DCCRN‑family model into a small yet powerful student. The authors introduce Distil‑DCCRN, a lightweight TF‑domain SE model trained via AT‑KL knowledge distillation from a stronger Uformer teacher, transferring both outputs and misaligned intermediate features across time and channel dimensions. The method combines time and channel dimension attention transfer with KL divergence and a balanced SI‑SNR loss, enabling effective cross‑model knowledge transfer despite differing STFT configurations. On the DNS dataset, Distil‑DCCRN achieves DNSMOS comparable to DCCRN while using only 1.1M parameters (about 30% of DCCRN) and delivering improved PESQ and SI‑SNR, with FLOPs roughly halved, demonstrating the practicality and effectiveness of AT‑KL KD for lightweight speech enhancement. Overall, the approach enables small, simple architectures to rival state‑of‑the‑art lightweight models, extending the reach of strong SE performance to parameter‑limited applications.

Abstract

The deep complex convolution recurrent network (DCCRN) achieves excellent speech enhancement performance by utilizing the audio spectrum's complex features. However, it has a large number of model parameters. We propose a smaller model, Distil-DCCRN, which has only 30% of the parameters compared to the DCCRN. To ensure that the performance of Distil-DCCRN matches that of the DCCRN, we employ the knowledge distillation (KD) method to use a larger teacher model to help train a smaller student model. We design a knowledge distillation (KD) method, integrating attention transfer and Kullback-Leibler divergence (AT-KL) to train the student model Distil-DCCRN. Additionally, we use a model with better performance and a more complicated structure, Uformer, as the teacher model. Unlike previous KD approaches that mainly focus on model outputs, our method also leverages the intermediate features from the models' middle layers, facilitating rich knowledge transfer across different structured models despite variations in layer configurations and discrepancies in the channel and time dimensions of intermediate features. Employing our AT-KL approach, Distil-DCCRN outperforms DCCRN as well as several other competitive models in both PESQ and SI-SNR metrics on the DNS test set and achieves comparable results to DCCRN in DNSMOS.
Paper Structure (14 sections, 8 equations, 1 figure, 2 tables)

This paper contains 14 sections, 8 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Overall Structure
  4. Attention Transfer
  5. Time Dimension
  6. Channel Dimension
  7. AT-KL
  8. Knowledge Distillation Method
  9. Experimental Results and Discussion
  10. Datasets
  11. Training Setups
  12. Experimental Results and Discussion
  13. Ablation study
  14. Conclusion

Figures (1)

  • Figure 1: The overall structure of our proposed AT-KL knowledge distillation method. We use Uformer uformer as the teacher model and Distil-DCCRN as the student model. For STFT configuration, their hop lengths are $\mu$ and $\nu$, respectively.