FastWhisper: Adaptive Self-knowledge Distillation for Real-time Automatic Speech Recognition

TL;DR

The paper addresses the challenge that conventional knowledge distillation can cause overreliance on the teacher, limiting generalization in automatic speech recognition under real-time constraints. It introduces adaptive self-knowledge distillation (ASKD), a two-stage training framework combining adaptive knowledge distillation (AKD) with a decaying weight $\alpha_{\text{AKD}}^e$ and adaptive self-knowledge distillation (SKD) using soft labels from the previous epoch ($P_T$), with losses $\mathcal{L}_{\text{AKD}} = \alpha_{\text{AKD}}^{e} \mathcal{L}_{\text{KL}}(P_S,P_T)$ and $\mathcal{L}_{\text{SKD}} = \mathcal{L}_{\text{CE}}((1 - \alpha_{\text{SKD}}^e) y + \alpha_{\text{SKD}}^e P_T, P_S)$. The method yields FastWhisper, a Whisper-encoder–based, compressed ASR model that achieves about $0.97\%$ lower WER than the teacher with significantly reduced latency (roughly $5\times$ faster) and good generalization to unseen data, while operating with fewer parameters. This approach offers a practical path to deploy accurate, real-time ASR in resource-constrained settings and demonstrates strong generalization across diverse datasets, with future work extending to multilingual scenarios.

Abstract

Knowledge distillation is one of the most effective methods for model compression. Previous studies have focused on the student model effectively training the predictive distribution of the teacher model. However, during training, the student model may inherit the shortcomings of the teacher model, which can lead to a decline in generalization capacity. To mitigate this issue, we propose adaptive self-knowledge distillation (ASKD), which dynamically reduces the dependence of the teacher model to improve the self-training capacity, and performs the self-knowledge distillation method to improve the generalization capacity of the student model. We further distill the Whisper model into a smaller variant, called FastWhisper. In our post-training setting, FastWhisper achieved a word error rate of 1.07% lower than the teacher model Whisper, and its relative inference time was 5 times faster.

Figures (2)

• Figure 1: Flow-chart of the adaptive self-knowledge distillation (ASKD). In adaptive knowledge distillation $\mathcal{L}_\text{AKD}$ denotes the knowledge distillation loss, $\mathcal{L}_\text{CE}$ represents the cross-entropy loss, and $\alpha_\text{AKD}^e$ indicates the adaptive knowledge distillation weight at timestamp ($e$) of the epoch. The $\lambda$ serves as the threshold for $\alpha_\text{AKD}^e$. In self-knowledge distillation $\mathcal{L}_\text{SKD}$ denotes the self-knowledge distillation loss, $\alpha_\text{SKD}^e$ represents the soften label weight for self-knowledge distillation, and $P_T$ indicates the predictive distribution of teacher model.
• Figure 2: WER results according to $\alpha_{\text{AKD}}$ value.