Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models

TL;DR

This work develops a novel approach, Importance-aware Sparse Tuning (IST), to fully utilize the inherent sparsity and select the most important subset of full layers with effective layer-wise importance scoring.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods have gained significant popularity for adapting pre-trained Large Language Models (LLMs) to downstream tasks, primarily due to their potential to significantly reduce memory and computational overheads. However, a common limitation in most PEFT approaches is their application of a uniform architectural design across all layers. This uniformity involves identical trainable modules and ignores the varying importance of each layer, leading to sub-optimal fine-tuning results. To overcome the above limitation and obtain better performance, we develop a novel approach, Importance-aware Sparse Tuning (IST), to fully utilize the inherent sparsity and select the most important subset of full layers with effective layer-wise importance scoring. The proposed IST is a versatile and plug-and-play technique compatible with various PEFT methods that operate on a per-layer basis. By leveraging the estimated importance scores, IST dynamically updates these selected layers in PEFT modules, leading to reduced memory demands. We further provide theoretical proof of convergence and empirical evidence of superior performance to demonstrate the advantages of IST over uniform updating strategies. Extensive experiments on a range of LLMs, PEFTs, and downstream tasks substantiate the effectiveness of our proposed method, showcasing IST's capacity to enhance existing layer-based PEFT methods. Our code is available at https://github.com/Kaiseem/IST.

Figures (5)

• Figure 1: (Left) Memory consumption of tuning a LLaMA 7B model with a token batch size of 1024 on a single device. Details refer to \ref{['sec:mem']}. (Right) In comparison to the vanilla tuning of all 32 and random tuning of 8 LoRA layers, IST achieves a better validation loss.
• Figure 2: Illustration of layer redundancy in PEFT training on the OPT-1.3B. (a) A greedy selection strategy is employed to iteratively remove the trained LoRA modules from the model. (b) Specific layers of the model are selectively fine-tuned using LoRA. The importance of layers depends on their contribution to the performance.
• Figure 3: Workflow of Importance-aware Sparse Tuning (IST): IST consists of two main loops: a fine-tuning loop, which selects a subset of layers for updating PEFT modules, and an importance updating loop, which estimates layer-wise importance by assessing the response suppression of the selected PEFT modules.
• Figure 4: Layer-wise importance on different tasks.
• Figure 5: Most existing PEFT approaches employ a layer-based design, consistently adding learnable modules or parameters to each layer of the transformer modules, including the Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN).

Theorems & Definitions (3)

• Corollary 3.1
• Lemma 3.2
• Corollary 3.3