Not All LoRA Parameters Are Essential: Insights on Inference Necessity

TL;DR

This work analyzes the layer-wise impact of LoRA fine-tuning in large language models and discovers a boundary layer separating information extraction from answer refinement, with bottom LoRA being essential for understanding and top LoRA largely redundant during inference. It proposes two boundary-detection strategies—manual probability-curve analysis and automated boundary search—to prune LoRA above the boundary without retraining, after an initial full LoRA fine-tuning. Through experiments on four generation tasks and three strong baselines, the approach yields consistent gains in generation quality and efficiency, while revealing varying effects on task-specific metrics and robustness to domain shifts. The findings offer a practical path to improve inference efficiency in LoRA-tuned LLMs and point to future work on automating boundary determination and extending to broader architectures.

Abstract

Current research on LoRA primarily focuses on minimizing the number of fine-tuned parameters or optimizing its architecture. However, the necessity of all fine-tuned LoRA layers during inference remains underexplored. In this paper, we investigate the contribution of each LoRA layer to the model's ability to predict the ground truth and hypothesize that lower-layer LoRA modules play a more critical role in model reasoning and understanding. To address this, we propose a simple yet effective method to enhance the performance of large language models (LLMs) fine-tuned with LoRA. Specifically, we identify a ``boundary layer'' that distinguishes essential LoRA layers by analyzing a small set of validation samples. During inference, we drop all LoRA layers beyond this boundary. We evaluate our approach on three strong baselines across four widely-used text generation datasets. Our results demonstrate consistent and significant improvements, underscoring the effectiveness of selectively retaining critical LoRA layers during inference.

Figures (6)

• Figure 1: The average maximum probability of the first four tokens for each layer of Llama3.1-8B-Instruct model fine-tuned with LoRA on the four datasets.
• Figure 2: The average maximum probability of the first four tokens for each layer of Llama3.1-8B-Instruct model fine-tuned with LoRA on the HotpotQA dataset while dropping specific LoRA layers during inference.
• Figure 3: The overview of our proposed method.
• Figure 4: The performance of different "boundary layer" of Llama3.1-8B-Instruct model. The Score means the corresponding automatic evaluation metric of four datasets.
• Figure 5: The probability difference of ground truth between our method and baseline on three datasets.