Distilling a speech and music encoder with task arithmetic

TL;DR

The paper addresses the challenge of building a unified speech+music SSL representation without prohibitive pre-training costs. It introduces task arithmetic for knowledge distillation: independently distill speech and music from domain-specific teachers, compute task vectors $\boldsymbol{\theta}_S$ and $\boldsymbol{\theta}_M$, and form a unified model via $\boldsymbol{\theta}_{S+M} = \boldsymbol{\theta}_0 + \lambda_1 \boldsymbol{\theta}_S + \lambda_2 \boldsymbol{\theta}_M$ with $\lambda_1+\lambda_2=1$. This approach reduces memory overhead and interference while enabling flexible domain emphasis through interpolation weights. Empirical results show Task Arithmetic can surpass ensemble distillation on several speech and music tasks and yields a higher SUPERB score, suggesting practical utility for ALLMs and cross-domain audio understanding. The work points to future improvements in interpolation strategies and parameter alignment to further enhance cross-domain generalization.

Abstract

Despite the progress in self-supervised learning (SSL) for speech and music, existing models treat these domains separately, limiting their capacity for unified audio understanding. A unified model is desirable for applications that require general representations, e.g. audio large language models. Nonetheless, directly training a general model for speech and music is computationally expensive. Knowledge Distillation of teacher ensembles may be a natural solution, but we posit that decoupling the distillation of the speech and music SSL models allows for more flexibility. Thus, we propose to learn distilled task vectors and then linearly interpolate them to form a unified speech+music model. This strategy enables flexible domain emphasis through adjustable weights and is also simpler to train. Experiments on speech and music benchmarks demonstrate that our method yields superior overall performance compared to ensemble distillation.

Figures (2)

• Figure 1: Overview of the distillation method via task arithmetic.
• Figure 2: Performance analysis of linear interpolation of distilled models under different interpolation weights.