SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters

TL;DR

The paper tackles the inefficiency of adapter-based fine-tuning by reinterpreting adapters as a pruning problem and introducing SparseAdapter, a pruning-at-initialization approach that preserves parameter budgets. By evaluating five pruning methods, notably SNIP and GraSP, SparseAdapter maintains or improves performance up to $80\%$ sparsity and can surpass full fine-tuning under a Large-Sparse setting. Across GLUE, XSum, and SQuAD with multiple backbones, SparseAdapter demonstrates robustness and compatibility with various adapter variants, achieving gains with the same or fewer trainable parameters. The authors release code and highlight the potential of SparseAdapter as a practical standard for parameter-efficient transfer learning in NLP.

Abstract

Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, becomes an appealing efficient alternative to the full model fine-tuning. Although computationally efficient, the recent Adapters often increase parameters (e.g. bottleneck dimension) for matching the performance of full model fine-tuning, which we argue goes against their original intention. In this work, we re-examine the parameter-efficiency of Adapters through the lens of network pruning (we name such plug-in concept as \texttt{SparseAdapter}) and find that SparseAdapter can achieve comparable or better performance than standard Adapters when the sparse ratio reaches up to 80\%. Based on our findings, we introduce an easy but effective setting ``\textit{Large-Sparse}'' to improve the model capacity of Adapters under the same parameter budget. Experiments on five competitive Adapters upon three advanced PLMs show that with proper sparse method (e.g. SNIP) and ratio (e.g. 40\%) SparseAdapter can consistently outperform their corresponding counterpart. Encouragingly, with the \textit{Large-Sparse} setting, we can obtain further appealing gains, even outperforming the full fine-tuning by a large margin. Our code will be released at: https://github.com/Shwai-He/SparseAdapter.

Figures (4)

• Figure 1: Performance of different parameter-efficient tuning methods on tasks from GLUE benchmark with RoBERTa-base encoder. We report the performance of Houlsby Adapters, Pfeiffer Adapters, LoRA as well as that used in our plug-in method SparseAdapter, where we denoted the normal sparse (in Table \ref{['main_results']} and \ref{['fig:variants']}) as "$\mathcal{S}$-" and Large-Sparse (in Table \ref{['fig:large_sparse']}) as "$\mathcal{LS}$-" in prefix.
• Figure 2: The comparison between randomly pruned adapters and standard adapters on datasets from GLUE.
• Figure 3: Schematic comparison of (a) standard adapter and (b) our proposed SparseAdapter.
• Figure 4: The comparison between SparseAdapters with Large-Sparse setting and standard adapters.