FISH-Tuning: Enhancing PEFT Methods with Fisher Information

TL;DR

FISH-Tuning introduces Fisher Information-based parameter selection (FISH Mask) into both addition-based and reparameterization-based PEFT methods to selectively update only the most impactful newly added parameters. By ranking parameters with empirical Fisher information and applying a gradient mask, it achieves superior performance compared with vanilla PEFT at the same trainable-parameter budget, across diverse datasets and models. The approach preserves training and inference efficiency while offering insights into Fisher-information-guided optimization for PEFT. Limitations include reduced effectiveness with UniPELT, and future work envisions extending to vision, multi-modality, and quantization contexts.

Abstract

The rapid growth in the parameter size of Large Language Models (LLMs) has spurred the development of Parameter-Efficient Fine-Tuning (PEFT) methods to mitigate the substantial computational costs of fine-tuning. Among these, Fisher Induced Sparse uncHanging (FISH) Mask is a selection-based PEFT technique that identifies a critical subset of pre-trained parameters using approximate Fisher information. While addition-based and reparameterization-based PEFT methods like LoRA and Adapter already fine-tune only a small number of parameters, the newly introduced parameters within these methods themselves present an opportunity for further optimization. Selectively fine-tuning only the most impactful among these new parameters could further reduce resource consumption while maintaining, or even improving, fine-tuning effectiveness. In this paper, we propose \textbf{FISH-Tuning}, a novel approach that incorporates FISH Mask into such PEFT methods, including LoRA, Adapter, and their variants. By leveraging Fisher information to identify and update only the most significant parameters within these added or reparameterized components, FISH-Tuning aims to achieve superior performance without increasing training time or inference latency compared to the vanilla PEFT methods. Experimental results across various datasets and pre-trained models demonstrate that FISH-Tuning consistently outperforms the vanilla PEFT methods when using the same proportion of trainable parameters. Code is available at https://anonymous.4open.science/r/FISH-Tuning-6F7C.

Figures (12)

• Figure 1: Original FISH Mask method (left) and original LoRA method (mid) and our FISH-Tuning method (right).
• Figure 2: Original LoRA method (left) and the LoRA-FISH method (right).
• Figure 3: Original Adapter method (left) and the Adapter-FISH method (right).
• Figure 4: Original FISH Mask method (left) and our FISH-Tuning method (right). The specific details of each dataset result of the right picture are represented in Appendix \ref{['sec:appendix']} Table \ref{['tab:lorarandandrevex2whole']}.
• Figure 5: Performance of FISH-Tuning method in different proportion of Trainable Parameters on the GSM8K dataset using Qwen2.5-7B model after 4 training epochs.