Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation

Swarup Ranjan Behera, Abhishek Dhiman, Karthik Gowda, Aalekhya Satya Narayani

TL;DR

FastAST tackles the efficiency bottleneck of Audio Spectrogram Transformer (AST) by integrating Token Merging (ToMe) to reduce token counts and speed up both inference and training. To preserve accuracy, it further employs Cross-Model Knowledge Distillation (CMKD) using EfficientNet-B2 and PaSST as teachers. Empirical results on ESC-50, Speech Commands V2, and Balanced AudioSet show substantial throughput gains with modest accuracy loss, and CMKD with CNN teachers offers the strongest improvements. These contributions enable near real-time, resource-efficient audio classification without requiring extensive retraining.

Abstract

Audio classification models, particularly the Audio Spectrogram Transformer (AST), play a crucial role in efficient audio analysis. However, optimizing their efficiency without compromising accuracy remains a challenge. In this paper, we introduce FastAST, a framework that integrates Token Merging (ToMe) into the AST framework. FastAST enhances inference speed without requiring extensive retraining by merging similar tokens in audio spectrograms. Furthermore, during training, FastAST brings about significant speed improvements. The experiments indicate that FastAST can increase audio classification throughput with minimal impact on accuracy. To mitigate the accuracy impact, we integrate Cross-Model Knowledge Distillation (CMKD) into the FastAST framework. Integrating ToMe and CMKD into AST results in improved accuracy compared to AST while maintaining faster inference speeds. FastAST represents a step towards real-time, resource-efficient audio analysis.

FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation

TL;DR

FastAST tackles the efficiency bottleneck of Audio Spectrogram Transformer (AST) by integrating Token Merging (ToMe) to reduce token counts and speed up both inference and training. To preserve accuracy, it further employs Cross-Model Knowledge Distillation (CMKD) using EfficientNet-B2 and PaSST as teachers. Empirical results on ESC-50, Speech Commands V2, and Balanced AudioSet show substantial throughput gains with modest accuracy loss, and CMKD with CNN teachers offers the strongest improvements. These contributions enable near real-time, resource-efficient audio classification without requiring extensive retraining.

Abstract

Audio classification models, particularly the Audio Spectrogram Transformer (AST), play a crucial role in efficient audio analysis. However, optimizing their efficiency without compromising accuracy remains a challenge. In this paper, we introduce FastAST, a framework that integrates Token Merging (ToMe) into the AST framework. FastAST enhances inference speed without requiring extensive retraining by merging similar tokens in audio spectrograms. Furthermore, during training, FastAST brings about significant speed improvements. The experiments indicate that FastAST can increase audio classification throughput with minimal impact on accuracy. To mitigate the accuracy impact, we integrate Cross-Model Knowledge Distillation (CMKD) into the FastAST framework. Integrating ToMe and CMKD into AST results in improved accuracy compared to AST while maintaining faster inference speeds. FastAST represents a step towards real-time, resource-efficient audio analysis.
Paper Structure (11 sections, 1 equation, 1 figure, 2 tables)

This paper contains 11 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. FastAST
  3. Audio Spectrogram Transformer (AST)
  4. Token Merging (ToMe)
  5. Cross-Model Knowledge Distillation (CMKD)
  6. Experimental Results and Analysis
  7. FastAST (ToMe)
  8. FastAST (Inf)
  9. FastAST (Train + Inf)
  10. FastAST (ToMe+CMKD)
  11. Conclusion

Figures (1)

  • Figure 1: Visualization of the proposed framework, FastAST, illustrating the integration of Token Merging (ToMe) and Cross-Model Knowledge Distillation (CMKD) into AST in the LHS box. The middle box demonstrates the incorporation of ToMe in the transformer encoder of AST, while the RHS box outlines the steps of the ToMe process. On the LHS box, the CMKD block depicts the process of cross-model knowledge distillation where a CNN, EfficientNet, serves as the teacher model.