Integrated Multi-Level Knowledge Distillation for Enhanced Speaker Verification

TL;DR

This work tackles the mismatch between image-based knowledge distillation and audio-based speaker verification by introducing Integrated Multi-level Knowledge Distillation (IML-KD). It fuses an Integrated Gradients–driven Integrated Inputs module with a three-level (instance, class, batch) knowledge distillation objective, enabling the student to mimic the teacher's temporal-context processing across varying speech durations. On VoxCeleb1, IML-KD achieves state-of-the-art SV performance, with $\text{EER}$ reduced to around $3.02\%$ and $\text{minDCF}$ down to approximately $0.316$ in one setting, outperforming vanilla KD, embedding-based KD, and prior multi-level KD methods. The approach is complemented by saliency-map analyses and deletion/insertion tests, demonstrating improved temporal fidelity and interpretability, with practical implications for compact, efficient speaker verification systems and broader audio tasks requiring temporal context distillation.

Abstract

Knowledge distillation (KD) is widely used in audio tasks, such as speaker verification (SV), by transferring knowledge from a well-trained large model (the teacher) to a smaller, more compact model (the student) for efficiency and portability. Existing KD methods for SV often mirror those used in image processing, focusing on approximating predicted probabilities and hidden representations. However, these methods fail to account for the multi-level temporal properties of speech audio. In this paper, we propose a novel KD method, i.e., Integrated Multi-level Knowledge Distillation (IML-KD), to transfer knowledge of various temporal-scale features of speech from a teacher model to a student model. In the IML-KD, temporal context information from the teacher model is integrated into novel Integrated Gradient-based input-sensitive representations from speech segments with various durations, and the student model is trained to infer these representations with multi-level alignment for the output. We conduct SV experiments on the VoxCeleb1 dataset to evaluate the proposed method. Experimental results demonstrate that IML-KD significantly enhances KD performance, reducing the Equal Error Rate (EER) by 5%.

Figures (3)

• Figure 1: Integrated Multi-level Knowledge Distillation for Speaker Verification.
• Figure 2: Visulization for utterance 'id10001-1zcIwhmdeo4-00001' in VoxCeleb1 dev.
• Figure 3: The deletion and insertion curve for student and teacher models.