Triage knowledge distillation for speaker verification

TL;DR

This work tackles the challenge of deploying speaker verification on devices with limited compute by refining knowledge distillation. It introduces Triage KD (TRKD), which partitions teacher posteriors into a target, a high-probability confusion-set, and a background-set using a cumulative cutoff $τ$, and learns via a three-mass KL objective plus a confusion-set conditional KL, while discarding background guidance; a $τ$-based curriculum gradually narrows the confusion-set to focus on the hardest impostors. Empirical results on VoxCeleb1/2 show TRKD consistently outperforms prior KD variants across homogeneous and heterogeneous teacher–student pairs, achieving the lowest EERs and a substantial average improvement over the non-KD baseline. The findings highlight the value of targeted, difficulty-aware supervision and curriculum scheduling in distillation, with practical implications for on-device SV and potential applicability to other domains.

Abstract

Deploying speaker verification on resource-constrained devices remains challenging due to the computational cost of high-capacity models; knowledge distillation (KD) offers a remedy. Classical KD entangles target confidence with non-target structure in a Kullback-Leibler term, limiting the transfer of relational information. Decoupled KD separates these signals into target and non-target terms, yet treats non-targets uniformly and remains vulnerable to the long tail of low-probability classes in large-class settings. We introduce Triage KD (TRKD), a distillation scheme that operationalizes assess-prioritize-focus. TRKD introduces a cumulative-probability cutoff $τ$ to assess per-example difficulty and partition the teacher posterior into three groups: the target class, a high-probability non-target confusion-set, and a background-set. To prioritize informative signals, TRKD distills the confusion-set conditional distribution and discards the background. Concurrently, it transfers a three-mass (target/confusion/background) that capture sample difficulty and inter-class confusion. Finally, TRKD focuses learning via a curriculum on $τ$: training begins with a larger $τ$ to convey broad non-target context, then $τ$ is progressively decreased to shrink the confusion-set, concentrating supervision on the most confusable classes. In extensive experiments on VoxCeleb1 with both homogeneous and heterogeneous teacher-student pairs, TRKD was consistently superior to recent KD variants and attained the lowest EER across all protocols.

Figures (2)

• Figure 1: Comparison: KD vs. DKD vs. TRKD.(a) Classical KD aligns teacher and student posteriors via $\mathcal{L}_{\mathrm{KD}}=\mathrm{KL}(\boldsymbol{p}^{\mathrm t}\,\|\,\boldsymbol{p}^{\mathrm s})$. (b) DKD decouples KD into a target--non-target KL term, $\mathcal{L}_{\mathrm{TCKD}}=\mathrm{KL}([p^{\mathrm t}_y,p^{\mathrm t}_{\setminus y}]\,\|\, [p^{\mathrm s}_y,p^{\mathrm s}_{\setminus y}])$, and a normalized non-target KL term, $\mathcal{L}_{\mathrm{NCKD}}=\mathrm{KL}(\tilde{\boldsymbol{p}}^{\mathrm t}\,\|\,\tilde{\boldsymbol{p}}^{\mathrm s})$, enhancing transfer among non-target classes. (c) TRKD partitions probabilities into three masses (target $y$, a high-probability non-target confusion-set $\mathcal{F}$, and a low-probability non-target background-set $\mathcal{B}$) and shrinks the confusion-set via a cumulative-probability cutoff $\tau(k)$ that decreases during training. The student minimizes a three-mass KL, $\mathcal{L}_{\mathrm{TMKD}}=\mathrm{KL}([p^{\mathrm t}_y,p^{\mathrm t}_{\mathcal{F}},p^{\mathrm t}_{\mathcal{B}}]\,\|\, [p^{\mathrm s}_y,p^{\mathrm s}_{\mathcal{F}},p^{\mathrm s}_{\mathcal{B}}])$, with a confusion-set conditional KL, $\mathcal{L}_{\mathrm{CFKD}}=\mathrm{KL}(\hat{\boldsymbol p}^{\mathrm t}_{\mathcal{F}}\,\|\,\hat{\boldsymbol p}^{\mathrm s}_{\mathcal{F}})$; the background-set term $\mathcal{L}_{\mathrm{BGKD}}$ is discarded to suppress long-tail noise.
• Figure 2: EER (%) versus student parameter count (log scale), comparing TRKD to the best prior approach (the minimum of DKD and GKD) across different student model sizes. The teacher is fixed at ReDimNet-B5 (dash--dot horizontal line; star marker). Students are drawn from two architecture families: ECAPA-TDNN (triangle markers) and ResNet (square markers), ordered left-to-right by increasing parameter count (ECAPA128$\to$ECAPA400$\to$ECAPA512$\to$ECAPA1024; RN18$\to$RN34$\to$RN50$\to$RN101). Labels on the TRKD curve (solid line) indicate each student’s EER and its relative improvement $\Delta$ (%) over the best prior (dashed line); the shaded region denotes this gain.