Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DISPATCH: Distilling Selective Patches for Speech Enhancement

Dohwan Kim, Jung-Woo Choi

TL;DR

This work tackles inefficiencies in knowledge distillation for speech enhancement by introducing DISPatch, which selectively distills only from spectrogram patches where the teacher offers meaningful guidance, as quantified by the Knowledge Gap Score $KGS_p = E_p^S - E_p^T$. It further extends this approach with MSSP, a frequency-dependent strategy that uses different patch sizes for low and high-frequency bands, and integrates these ideas with existing KD methods, including DFKD. Empirical results on ConvTasNet and DCCRN-CL show consistent gains over baselines, with the strongest improvements achieved when combining DISPatch and MSSP inside a frequency-aware KD framework like DFKD, notably using a 10/40 patch configuration. The proposed selective transfer approach yields practical benefits for on-device speech enhancement by reducing reliance on low-quality teacher regions and focusing learning on the most impactful patches.

Abstract

In speech enhancement, knowledge distillation (KD) compresses models by transferring a high-capacity teacher's knowledge to a compact student. However, conventional KD methods train the student to mimic the teacher's output entirely, which forces the student to imitate the regions where the teacher performs poorly and to apply distillation to the regions where the student already performs well, which yields only marginal gains. We propose Distilling Selective Patches (DISPatch), a KD framework for speech enhancement that applies the distillation loss to spectrogram patches where the teacher outperforms the student, as determined by a Knowledge Gap Score. This approach guides optimization toward areas with the most significant potential for student improvement while minimizing the influence of regions where the teacher may provide unreliable instruction. Furthermore, we introduce Multi-Scale Selective Patches (MSSP), a frequency-dependent method that uses different patch sizes across low- and high-frequency bands to account for spectral heterogeneity. We incorporate DISPatch into conventional KD methods and observe consistent gains in compact students. Moreover, integrating DISPatch and MSSP into a state-of-the-art frequency-dependent KD method considerably improves performance across all metrics.

DISPATCH: Distilling Selective Patches for Speech Enhancement

TL;DR

This work tackles inefficiencies in knowledge distillation for speech enhancement by introducing DISPatch, which selectively distills only from spectrogram patches where the teacher offers meaningful guidance, as quantified by the Knowledge Gap Score . It further extends this approach with MSSP, a frequency-dependent strategy that uses different patch sizes for low and high-frequency bands, and integrates these ideas with existing KD methods, including DFKD. Empirical results on ConvTasNet and DCCRN-CL show consistent gains over baselines, with the strongest improvements achieved when combining DISPatch and MSSP inside a frequency-aware KD framework like DFKD, notably using a 10/40 patch configuration. The proposed selective transfer approach yields practical benefits for on-device speech enhancement by reducing reliance on low-quality teacher regions and focusing learning on the most impactful patches.

Abstract

In speech enhancement, knowledge distillation (KD) compresses models by transferring a high-capacity teacher's knowledge to a compact student. However, conventional KD methods train the student to mimic the teacher's output entirely, which forces the student to imitate the regions where the teacher performs poorly and to apply distillation to the regions where the student already performs well, which yields only marginal gains. We propose Distilling Selective Patches (DISPatch), a KD framework for speech enhancement that applies the distillation loss to spectrogram patches where the teacher outperforms the student, as determined by a Knowledge Gap Score. This approach guides optimization toward areas with the most significant potential for student improvement while minimizing the influence of regions where the teacher may provide unreliable instruction. Furthermore, we introduce Multi-Scale Selective Patches (MSSP), a frequency-dependent method that uses different patch sizes across low- and high-frequency bands to account for spectral heterogeneity. We incorporate DISPatch into conventional KD methods and observe consistent gains in compact students. Moreover, integrating DISPatch and MSSP into a state-of-the-art frequency-dependent KD method considerably improves performance across all metrics.

Paper Structure

This paper contains 12 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Not All Spectrogram Regions Are Equal
  4. DISPatch: Distilling Selective Patches
  5. MSSP: Multi-Scale Selective Patches
  6. Training Setup with DISPatch
  7. EXPERIMENTS
  8. Datasets
  9. Implementation Details
  10. Experimental Results
  11. CONCLUSIONS
  12. Acknowledgements

Figures (2)

  • Figure 1: Overview of DISPatch framework. DISPatch framework separates spectrograms into patches and selectively applies KD loss to those exhibiting a high Knowledge Gap, where the teacher's error is low (L) and the student's is high (H). Patch size: $2C\times N \times1$ .
  • Figure 2: Analysis of patch-level teacher-student misalignment. (a) Confusion matrix of patch classification. (b) Scatter plot comparing patch-wise errors between the teacher and the student.