GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs

TL;DR

This work introduces GradPruner, a gradient-guided layer pruning method for LLMs that leverages the Initial Gradient Information Accumulation Matrix ($IGIA$-Matrix) computed from early LoRA fine-tuning steps to rank layer importance. To maximize pruning while preserving accuracy, GradPruner employs a two-stage strategy: (i) prune layers based on aggregated IGIA information, and (ii) merge pruned layers with retained ones using sign-based (and optionally Fisher) merging to reduce interference. Across two LLMs and eight downstream tasks, GradPruner achieves about 40% parameter reduction with only around 0.99% average drop in accuracy, while delivering notable training and inference efficiency gains compared to baselines. The method demonstrates robustness across datasets, scales from 1B to 13B models, and maintains performance under varying data sizes and adapter strategies, making it a practical approach for efficient downstream fine-tuning and inference of large models.

Abstract

Fine-tuning Large Language Models (LLMs) with downstream data is often considered time-consuming and expensive. Structured pruning methods are primarily employed to improve the inference efficiency of pre-trained models. Meanwhile, they often require additional time and memory for training, knowledge distillation, structure search, and other strategies, making efficient model fine-tuning challenging to achieve. To simultaneously enhance the training and inference efficiency of downstream task fine-tuning, we introduce GradPruner, which can prune layers of LLMs guided by gradients in the early stages of fine-tuning. GradPruner uses the cumulative gradients of each parameter during the initial phase of fine-tuning to compute the Initial Gradient Information Accumulation Matrix (IGIA-Matrix) to assess the importance of layers and perform pruning. We sparsify the pruned layers based on the IGIA-Matrix and merge them with the remaining layers. Only elements with the same sign are merged to reduce interference from sign variations. We conducted extensive experiments on two LLMs across eight downstream datasets. Including medical, financial, and general benchmark tasks. The results demonstrate that GradPruner has achieved a parameter reduction of 40% with only a 0.99% decrease in accuracy. Our code is publicly available.

Figures (9)

• Figure 1: The loss of training while LoRA fine-tuning PubMedQA (a) and PIQA (b) on Llama3.1-8B. The loss value showed a rapid decrease during the initial training steps.
• Figure 2: Gradient Sensitivity Analysis of the IGIA-Matrix on Llama3.1-8B. We can see that the layer importance measured at such an early stage can accurately reflect the results after the entire training process.
• Figure 3: The framework diagram of Parameter Importance Evaluation and Layer Pruning in GradPruner. The first step involves obtaining gradients through a small amount of LoRA fine-tuning. The second step calculates the IGIA-Matrix based on gradients. In the third step, we assess the importance of each parameter and each layer based on the IGIA-Matrix and subsequently prune the layers accordingly.
• Figure 4: The framework diagram of Layer Merging in GradPruner. The first step is to sparsify the pruned modules using the IGIA-Matrix as the criterion. The second step is to merge the pruned layer with the preceding retained layer based on their signs.
• Figure 5: Experimental results of performing Only the layer pruning in GradPruner. We report the average accuracy across eight datasets.