Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters

Umberto Cappellazzo, Daniele Falavigna, Alessio Brutti

TL;DR

This work addresses parameter-efficient fine-tuning for audio models by introducing Soft-Mixture of Adapters (Soft-MoA), which uses adapters as experts within a Soft MoE framework to scale the AST while keeping computation affordable. Soft-MoA achieves soft routing of input tokens to multiple adapters via dispatch and combine weights, enabling many adapters to participate without full dense computation. Across four audio benchmarks, Soft-MoA consistently outperforms single-adapter baselines and matches Dense-MoA in accuracy while reducing training cost, with the largest gains on the FSC dataset. Ablation studies show Soft-MoA scales well with more experts, avoids expert imbalance, and is most effective when using many adapters with few slots, indicating strong practical value for PETL in audio. Overall, Soft-MoA provides a principled, scalable, and efficient approach to adapting large audio transformers to downstream tasks.

Abstract

Mixture of Experts (MoE) architectures have recently started burgeoning due to their ability to scale model's capacity while maintaining the computational cost affordable. Furthermore, they can be applied to both Transformers and State Space Models, the current state-of-the-art models in numerous fields. While MoE has been mostly investigated for the pre-training stage, its use in parameter-efficient transfer learning settings is under-explored. To narrow this gap, this paper attempts to demystify the use of MoE for parameter-efficient fine-tuning of Audio Spectrogram Transformers to audio and speech downstream tasks. Specifically, we propose Soft Mixture of Adapters (Soft-MoA). It exploits adapters as the experts and, leveraging the recent Soft MoE method, it relies on a soft assignment between the input tokens and experts to keep the computational time limited. Extensive experiments across 4 benchmarks demonstrate that Soft-MoA outperforms the single adapter method and performs on par with the dense MoA counterpart. We finally present ablation studies on key elements of Soft-MoA, showing for example that Soft-MoA achieves better scaling with more experts, as well as ensuring that all experts contribute to the computation of the output tokens, thus dispensing with the expert imbalance issue.

Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters

TL;DR

This work addresses parameter-efficient fine-tuning for audio models by introducing Soft-Mixture of Adapters (Soft-MoA), which uses adapters as experts within a Soft MoE framework to scale the AST while keeping computation affordable. Soft-MoA achieves soft routing of input tokens to multiple adapters via dispatch and combine weights, enabling many adapters to participate without full dense computation. Across four audio benchmarks, Soft-MoA consistently outperforms single-adapter baselines and matches Dense-MoA in accuracy while reducing training cost, with the largest gains on the FSC dataset. Ablation studies show Soft-MoA scales well with more experts, avoids expert imbalance, and is most effective when using many adapters with few slots, indicating strong practical value for PETL in audio. Overall, Soft-MoA provides a principled, scalable, and efficient approach to adapting large audio transformers to downstream tasks.

Abstract

Mixture of Experts (MoE) architectures have recently started burgeoning due to their ability to scale model's capacity while maintaining the computational cost affordable. Furthermore, they can be applied to both Transformers and State Space Models, the current state-of-the-art models in numerous fields. While MoE has been mostly investigated for the pre-training stage, its use in parameter-efficient transfer learning settings is under-explored. To narrow this gap, this paper attempts to demystify the use of MoE for parameter-efficient fine-tuning of Audio Spectrogram Transformers to audio and speech downstream tasks. Specifically, we propose Soft Mixture of Adapters (Soft-MoA). It exploits adapters as the experts and, leveraging the recent Soft MoE method, it relies on a soft assignment between the input tokens and experts to keep the computational time limited. Extensive experiments across 4 benchmarks demonstrate that Soft-MoA outperforms the single adapter method and performs on par with the dense MoA counterpart. We finally present ablation studies on key elements of Soft-MoA, showing for example that Soft-MoA achieves better scaling with more experts, as well as ensuring that all experts contribute to the computation of the output tokens, thus dispensing with the expert imbalance issue.
Paper Structure (11 sections, 4 equations, 3 figures, 3 tables)

This paper contains 11 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. AST model recap
  4. Adapters
  5. Dense and Soft MoA
  6. Experiments and Discussion
  7. Implementation Details
  8. Main Results and Discussion
  9. Ablation Studies
  10. Conclusion
  11. Acknowledgments

Figures (3)

  • Figure 1: (a) For each AST layer, the Soft/Dense MoA blocks are inserted parallel to MHSA (Pfeiffer) or parallel to both MHSA and FFN sub-layers (Houlsby). (b) Illustration of Dense-MoA, whereby each expert contribution, scaled by the router's distribution (thickness of the arrows), is summed to produce the final output. (c) In Soft-MoA, each expert only processes a subset of slots (here $2$), and each slot accepts as input a weighted combination of all input tokens (thickness of the arrows). Note that the trainable parameters are represented by dashed blocks. Best viewed in color.
  • Figure 2: (Left). The accuracy trend as more parameters are used. (Middle). The effect of the number of adapters given a fixed parameters budget. (Right). Adapters contribution to the output tokens for various layers. Results reported for FSC.
  • Figure 3: Distribution of expert activation frequencies per class.