Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition
Qingran Yang, Botao Zhao, Zuheng Kang, Xue Li, Yayun He, Chuhang Liu, Xulong Zhang, Xiaoyang Qu, Junqing Peng, Jianzong Wang
TL;DR
This work tackles the high computational cost of Large Audio-Language Models for Speech Emotion Recognition by introducing PL-Distill, a two-pronged knowledge distillation framework. Projector-Level Distillation uses Attention-weighted Centered Kernel Alignment to align cross-modal audio embeddings despite dimensional differences and to focus on emotionally salient time steps, while Logits-Level Distillation aligns teacher and student outputs across audio and text modalities. The approach compresses an 8.4B-parameter teacher to a 1.1B-parameter student and consistently outperforms baselines and pretrained models across IEMOCAP, RAVDESS, and SAVEE, with ablations highlighting the importance of AwCKA and projector alignment. This work demonstrates the practicality of efficient cross-modal distillation for LALMs in SER and suggests broader applicability to other audio-language tasks via targeted, token-aware representation and logit distillation.
Abstract
The emergence of Large Audio-Language Models (LALMs) has advanced Speech Emotion Recognition (SER), but their size limits deployment in resource-constrained environments. While Knowledge Distillation is effective for LALM compression, existing methods remain underexplored in distilling the cross-modal projection module (Projector), and often struggle with alignment due to differences in feature dimensions. We propose PL-Distill, a KD framework that combines Projector-Level Distillation (PDist) to align audio embeddings and Logits-Level Distillation (LDist) to align output logits. PDist introduces Attention-weighted Centered Kernel Alignment, a novel approach we propose to highlight important time steps and address dimension mismatches. Meanwhile, LDist minimizes the Kullback-Leibler divergence between teacher and student logits from audio and text modalities. On IEMOCAP, RAVDESS, and SAVEE, PL-Distill compresses an 8.4B-parameter teacher to a compact 1.1B-parameter student, consistently outperforming the teacher, state-of-the-art pretrained models, and other KD baselines across all metrics.