Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Audio Captioning with Encoder-Level Knowledge Distillation

Xuenan Xu, Haohe Liu, Mengyue Wu, Wenwu Wang, Mark D. Plumbley

TL;DR

This work tackles the challenge of efficient automated audio captioning by introducing encoder-level knowledge distillation to compress an encoder–decoder AAC model. A large teacher (HTSAT-BART) guides a lightweight student (EfficientNet-B2 encoder + compact Transformer decoder) through a combined loss: $ abla = abla_{sup} + abla_{seq} + abla_{enc}$, with two encoder KD variants, $ abla_{contra}$ and $ abla_{mse}$. The results show that encoder-focused KD, particularly the contrastive variant, yields better performance in data-scarce conditions, and that incorporating audio-only data with pseudo captions further improves results, achieving near-teacher performance with ~19x faster inference and only a fraction of the teacher's parameters. This approach delivers a practical pathway to deploy efficient AAC systems on resource-constrained devices while preserving high captioning quality. The work also provides a clear comparative analysis between encoder vs. decoder bottlenecks and demonstrates the value of data augmentation via audio-only data for knowledge distillation in generation tasks.

Abstract

Significant improvement has been achieved in automated audio captioning (AAC) with recent models. However, these models have become increasingly large as their performance is enhanced. In this work, we propose a knowledge distillation (KD) framework for AAC. Our analysis shows that in the encoder-decoder based AAC models, it is more effective to distill knowledge into the encoder as compared with the decoder. To this end, we incorporate encoder-level KD loss into training, in addition to the standard supervised loss and sequence-level KD loss. We investigate two encoder-level KD methods, based on mean squared error (MSE) loss and contrastive loss, respectively. Experimental results demonstrate that contrastive KD is more robust than MSE KD, exhibiting superior performance in data-scarce situations. By leveraging audio-only data into training in the KD framework, our student model achieves competitive performance, with an inference speed that is 19 times faster\footnote{An online demo is available at \url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}}.

Efficient Audio Captioning with Encoder-Level Knowledge Distillation

TL;DR

This work tackles the challenge of efficient automated audio captioning by introducing encoder-level knowledge distillation to compress an encoder–decoder AAC model. A large teacher (HTSAT-BART) guides a lightweight student (EfficientNet-B2 encoder + compact Transformer decoder) through a combined loss: , with two encoder KD variants, and . The results show that encoder-focused KD, particularly the contrastive variant, yields better performance in data-scarce conditions, and that incorporating audio-only data with pseudo captions further improves results, achieving near-teacher performance with ~19x faster inference and only a fraction of the teacher's parameters. This approach delivers a practical pathway to deploy efficient AAC systems on resource-constrained devices while preserving high captioning quality. The work also provides a clear comparative analysis between encoder vs. decoder bottlenecks and demonstrates the value of data augmentation via audio-only data for knowledge distillation in generation tasks.

Abstract

Significant improvement has been achieved in automated audio captioning (AAC) with recent models. However, these models have become increasingly large as their performance is enhanced. In this work, we propose a knowledge distillation (KD) framework for AAC. Our analysis shows that in the encoder-decoder based AAC models, it is more effective to distill knowledge into the encoder as compared with the decoder. To this end, we incorporate encoder-level KD loss into training, in addition to the standard supervised loss and sequence-level KD loss. We investigate two encoder-level KD methods, based on mean squared error (MSE) loss and contrastive loss, respectively. Experimental results demonstrate that contrastive KD is more robust than MSE KD, exhibiting superior performance in data-scarce situations. By leveraging audio-only data into training in the KD framework, our student model achieves competitive performance, with an inference speed that is 19 times faster\footnote{An online demo is available at \url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}}.
Paper Structure (17 sections, 8 equations, 2 figures, 3 tables)

This paper contains 17 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Knowledge Distillation Framework
  3. Framework Overview
  4. Encoder-Level Knowledge Distillation
  5. Distillation via Contrastive Learning
  6. Distillation via Optimizing Mean Squared Error
  7. Training with Audio-Only Data
  8. Experimental Setup
  9. Dataset
  10. Hyper-parameters
  11. Evaluation Metrics
  12. Results
  13. Bottleneck Analysis
  14. Knowledge Distillation from the Teacher Model
  15. Inference Speedup
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The comparison of performance-size tradeoff between our model and previous methods, evaluated on AudioCaps.
  • Figure 2: An overview of our proposed audio captioning knowledge distillation framework, which combines supervised loss $\mathcal{L}_{sup}$, sequence-level distillation loss $\mathcal{L}_{seq}$ and encoder-level distillation loss $\mathcal{L}_{enc}$ for training. We explore two kinds of $\mathcal{L}_{enc}$: 1) contrastive loss; 2) MSE loss.