Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning

Qizhe Zhang, Bocheng Zou, Ruichuan An, Jiaming Liu, Shanghang Zhang

TL;DR

Mixture of Sparse Adapters, or MoSA, is proposed as a novel Adapter Tuning method to fully unleash the potential of each parameter in the adapter to achieve significantly better performance than standard adapters without any additional computational or storage overhead.

Abstract

With the rapid growth in the scale of pre-trained foundation models, parameter-efficient fine-tuning techniques have gained significant attention, among which Adapter Tuning is the most widely used. Despite achieving efficiency, it still underperforms full fine-tuning, and the performance improves at the cost of an increase in parameters. Recent efforts have either focused on training multiple adapter experts to increase model capacity or on pruning adapters to achieve parameter efficiency. However, both approaches introduce more parameters compared to the original adapter, hence are not computationally efficient. Motivated by this, we propose Mixture of Sparse Adapters, or MoSA, as a novel Adapter Tuning method to fully unleash the potential of each parameter in the adapter. We first split the standard adapter into multiple non-overlapping modules, then stochastically activate them for sparse training, and finally merge them to form a complete adapter after tuning. In this way, MoSA can achieve significantly better performance than standard adapters without any additional computational or storage overhead. Furthermore, we propose a hierarchical sparse strategy to better leverage limited training data. Extensive experiments on a series of 27 visual tasks demonstrate that MoSA consistently outperforms other Adapter Tuning methods as well as other baselines by a large margin. Furthermore, MoSA brings consistent improvements across various model scales, architectures, and different PEFT methods. Code will be released.

MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning

TL;DR

Mixture of Sparse Adapters, or MoSA, is proposed as a novel Adapter Tuning method to fully unleash the potential of each parameter in the adapter to achieve significantly better performance than standard adapters without any additional computational or storage overhead.

Abstract

With the rapid growth in the scale of pre-trained foundation models, parameter-efficient fine-tuning techniques have gained significant attention, among which Adapter Tuning is the most widely used. Despite achieving efficiency, it still underperforms full fine-tuning, and the performance improves at the cost of an increase in parameters. Recent efforts have either focused on training multiple adapter experts to increase model capacity or on pruning adapters to achieve parameter efficiency. However, both approaches introduce more parameters compared to the original adapter, hence are not computationally efficient. Motivated by this, we propose Mixture of Sparse Adapters, or MoSA, as a novel Adapter Tuning method to fully unleash the potential of each parameter in the adapter. We first split the standard adapter into multiple non-overlapping modules, then stochastically activate them for sparse training, and finally merge them to form a complete adapter after tuning. In this way, MoSA can achieve significantly better performance than standard adapters without any additional computational or storage overhead. Furthermore, we propose a hierarchical sparse strategy to better leverage limited training data. Extensive experiments on a series of 27 visual tasks demonstrate that MoSA consistently outperforms other Adapter Tuning methods as well as other baselines by a large margin. Furthermore, MoSA brings consistent improvements across various model scales, architectures, and different PEFT methods. Code will be released.
Paper Structure (17 sections, 8 equations, 5 figures, 14 tables)

This paper contains 17 sections, 8 equations, 5 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview
  5. Sparse Adapter Splitting
  6. Stochastic Activation Tuning
  7. Jigsaw-like Adapter Merging
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Extended Results on Different Backbone Scales and LoRA
  12. Ablation Study
  13. Visualization
  14. Conclusion
  15. Dataset Details
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Different Adapter Tuning diagrams: (a) The standard adapter simply inserts a bottleneck module into each Transformer layer. (b) Sparse Adapter. It prunes the standard adapter before tuning, updating only a small subset of the retained parameters. (c) Our proposed Mixture of Sparse Adapters (MoSA). We first split the standard adapter into multiple non-overlapping modules, then stochastically activate them for sparse training, and finally merge them to form a complete dense adapter.
  • Figure 2: Architecture design of Mixture of Sparse Adapters. In the training phase, MoSA stochastically activates a sparse adapter during each forward pass; in the inference phase, MoSA merges multiple sparse adapters into a complete one to enhance efficiency.
  • Figure 3: Impact of adapter bottleneck dimensions
  • Figure 4: t-SNE visualization on CIFAR-100
  • Figure 5: Per-task results on VTAB-1k with ViT-B/16 pre-trained on ImageNet-21K.