SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning

TL;DR

SIBO introduces a simple, plug-and-play booster for parameter-efficient fine-tuning by injecting an initial residual into PEFT modules at every Transformer layer. By preserving a portion of the original input representation via a tunable factor $\lambda$, SIBO mitigates over-smoothing that degrades deep PEFT performance and is compatible with both Adapters and LoRA. Across 22 benchmarks spanning arithmetic, commonsense, and GLUE, SIBO yields consistent gains, including up to 15.7% improvements on arithmetic and 23.5% on commonsense tasks, and in GLUE can match full-model fine-tuning for Adapter-SIBO. The method remains computationally lightweight and broadly applicable, with future work aimed at making $\lambda$ learnable to further reduce tuning overhead.

Abstract

Fine-tuning all parameters of large language models (LLMs) necessitates substantial computational power and extended time. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques, such as Adapter tuning and LoRA, allow for adjustments to only a minor fraction of the parameters of these LLMs. Concurrently, it has been noted that the issue of over-smoothing diminishes the effectiveness of these Transformer-based LLMs, resulting in suboptimal performances in downstream tasks. In this paper, we present SIBO, which is a SImple BOoster to enhance PEFT, by injecting an initial residual. SIBO is straightforward and readily extensible to a range of state-of-the-art PEFT techniques to alleviate over-smoothing and enhance performance. Extensive experiments on 22 benchmark datasets demonstrate that SIBO significantly enhances the performance of various strong baselines, achieving up to 15.7% and 23.5% improvement over existing PEFT methods on the arithmetic and commonsense reasoning tasks, respectively.

Figures (6)

• Figure 1: Over-smoothing in PEFT. The results are the averaged token-wise similarity of sentences in the test sets of the corpora in the GLUE benchmark wang2018glue, with BERT-large as the backbone.
• Figure 2: Over-smoothing in PEFT. The results are the averaged token-wise similarity of sentences in the test sets of MAWPS koncel2016mawps and SVAMP patel2021nlp, with LLaMA (13B) as the backbone.
• Figure 3: Proposed framework SIBO, applying to two popular PEFT methods: (a) Adapter, and (b) LoRA.
• Figure 4: Impact of initial residual portion $\lambda$.
• Figure 5: Token-wise similarity in last five layers computed from PEFT methods, with and without SIBO.