Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance

TL;DR

The paper addresses the challenge of deploying efficient speech denoising by improving knowledge distillation (KD) from a large teacher to a smaller student under latent-space dimensionality and ordering mismatches. It introduces a cosine-distance KD within a denoising autoencoder (DAE) that employs a linear inverted bottleneck to align teacher and student latent representations without forcing the student to mimic the teacher’s exact feature distribution, optimizing a joint loss $\mathcal{L}_{tot} = \lambda_{kd}\mathcal{L}_{kd} + \lambda_{out}\mathcal{L}_{out}$ where $\mathcal{L}_{kd}$ is the cosine distance between $\mathbf{H}_{\text{B},t}$ and $\mathbf{H}_{\text{E},s}$ and $\mathcal{L}_{out}$ is a supervised denoising loss such as SI-SNR. The method demonstrates improved robustness and performance over a state-of-the-art KD baseline on DNS-Challenge-derived data, particularly when dimensionalities are matched in the $(C,H)$ axes, and highlights practical implications for deploying lightweight denoisers on devices with limited resources. Overall, the work provides a scalable KD framework that leverages latent-space alignment through cosine similarity and linear bottlenecks to enable effective knowledge transfer across mismatched teacher–student architectures. The findings suggest potential for more reliable, hardware-friendly speech denoisers with modest gains in standard metrics, motivating further exploration of alternative similarity measures and training dynamics.

Abstract

Speech denoising is a generally adopted and impactful task, appearing in many common and everyday-life use cases. Although there are very powerful methods published, most of those are too complex for deployment in everyday and low-resources computational environments, like hand-held devices, intelligent glasses, hearing aids, etc. Knowledge distillation (KD) is a prominent way for alleviating this complexity mismatch and is based on the transferring/distilling of knowledge from a pre-trained complex model, the teacher, to another less complex one, the student. Existing KD methods for speech denoising are based on processes that potentially hamper the KD by bounding the learning of the student to the distribution, information ordering, and feature dimensionality learned by the teacher. In this paper, we present and assess a method that tries to treat this issue, by exploiting the well-known denoising-autoencoder framework, the linear inverted bottlenecks, and the properties of the cosine similarity. We use a public dataset and conduct repeated experiments with different mismatching scenarios between the teacher and the student, reporting the mean and standard deviation of the metrics of our method and another, state-of-the-art method that is used as a baseline. Our results show that with the proposed method, the student can perform better and can also retain greater mismatching conditions compared to the teacher.

Figures (3)

• Figure 1: Illustration of the DAE process, where $q(\cdot)$ is a corruption process that yields $\tilde{\mathbf{X}}$. With red color are indicated the noisy/corrupted data ($\tilde{\mathbf{X}}$), with black the clean ones ($\mathbf{X}$), with purple is the mapping that is learned from the DAE, and the green axis are for visual convenience. Image is after vincent:2010:jmlr.
• Figure 2: Illustration of the employed UNet architecture for the utilized DAEs. With $\odot$ is the Hadamard product
• Figure 3: Illustration of the KD process, where $\oplus$ is simple summation.