Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contrastive Knowledge Distillation for Embedding Refinement in Personalized Speech Enhancement

Thomas Serre, Mathieu Fontaine, Éric Benhaim, Slim Essid

TL;DR

The paper addresses the sensitivity of personalized speech enhancement to target-voice embeddings by enabling on-the-fly refinement with a tiny 150k-parameter encoder trained via contrastive knowledge distillation to mimic a heavy ECAPA-TDNN. This lightweight encoder is integrated into a PSE framework based on pDeepFilterNet2, using a similarity-based refinement that adapts the target representation during inference while keeping computational costs low. Key contributions include the contrastive KD training strategy, the lightweight TinyECAPA, and demonstrated gains on the DNS5 benchmark with favorable model efficiency compared to larger baselines. The proposed approach enables robust target voice extraction in dynamic inference conditions, with practical impact for low-resource deployment and privacy-conscious speech applications.

Abstract

Personalized speech enhancement (PSE) has shown convincing results when it comes to extracting a known target voice among interfering ones. The corresponding systems usually incorporate a representation of the target voice within the enhancement system, which is extracted from an enrollment clip of the target voice with upstream models. Those models are generally heavy as the speaker embedding's quality directly affects PSE performances. Yet, embeddings generated beforehand cannot account for the variations of the target voice during inference time. In this paper, we propose to perform on-thefly refinement of the speaker embedding using a tiny speaker encoder. We first introduce a novel contrastive knowledge distillation methodology in order to train a 150k-parameter encoder from complex embeddings. We then use this encoder within the enhancement system during inference and show that the proposed method greatly improves PSE performances while maintaining a low computational load.

Contrastive Knowledge Distillation for Embedding Refinement in Personalized Speech Enhancement

TL;DR

The paper addresses the sensitivity of personalized speech enhancement to target-voice embeddings by enabling on-the-fly refinement with a tiny 150k-parameter encoder trained via contrastive knowledge distillation to mimic a heavy ECAPA-TDNN. This lightweight encoder is integrated into a PSE framework based on pDeepFilterNet2, using a similarity-based refinement that adapts the target representation during inference while keeping computational costs low. Key contributions include the contrastive KD training strategy, the lightweight TinyECAPA, and demonstrated gains on the DNS5 benchmark with favorable model efficiency compared to larger baselines. The proposed approach enables robust target voice extraction in dynamic inference conditions, with practical impact for low-resource deployment and privacy-conscious speech applications.

Abstract

Personalized speech enhancement (PSE) has shown convincing results when it comes to extracting a known target voice among interfering ones. The corresponding systems usually incorporate a representation of the target voice within the enhancement system, which is extracted from an enrollment clip of the target voice with upstream models. Those models are generally heavy as the speaker embedding's quality directly affects PSE performances. Yet, embeddings generated beforehand cannot account for the variations of the target voice during inference time. In this paper, we propose to perform on-thefly refinement of the speaker embedding using a tiny speaker encoder. We first introduce a novel contrastive knowledge distillation methodology in order to train a 150k-parameter encoder from complex embeddings. We then use this encoder within the enhancement system during inference and show that the proposed method greatly improves PSE performances while maintaining a low computational load.
Paper Structure (14 sections, 1 equation, 3 figures, 1 table)

This paper contains 14 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Speaker embeddings
  4. Contrastive training strategy
  5. PSE framework: pDeepFilterNet2
  6. On-the-fly embedding refinement
  7. Experiments
  8. Speaker embedding training
  9. PSE training
  10. Results
  11. Speaker encoders
  12. PSE evaluation setup
  13. Discussions
  14. Conclusions

Figures (3)

  • Figure 1: (Left) Tiny encoder's architecture (Right) Methodology for on-the-fly similarity computation.
  • Figure 2: tSNE projection of 10 speakers' embeddings unseen during training, each represented by a color.
  • Figure 3: SIG vs BAK on Track1 of DNS5 Blind test set for different values of the scaling factor $\alpha$