The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models

TL;DR

The paper tackles the challenge of deploying large language models on resource-constrained devices by proposing Continuous Layer Pruning (CLP). CLP automatically identifies contiguous layer blocks to prune via a differentiable concave gating mechanism and a KL-divergence based pruning objective, followed by a fast recovery step called cutoff endpoint tuning that updates only the layers adjacent to the pruned block. The approach yields superior performance retention across multiple architectures and scales (7B–70B) and is compatible with post-training quantization, enabling substantial model compression with limited accuracy loss. This work provides a practical pathway for edge deployment of large language models, combining effective structural pruning with efficient fine-tuning strategies.

Abstract

Although large language models (LLMs) have achieved revolutionary breakthroughs in many fields, their large model size and high computational cost pose significant challenges for practical deployment on resource-constrained edge devices. To this end, layer pruning has been proposed to reduce the computational overhead by directly removing redundant layers. However, existing layer pruning methods typically rely on hand-crafted metrics to evaluate and remove individual layers, while ignoring the dependencies between layers. This can disrupt the model's information flow and severely degrade performance. To address these issues, we propose CLP, a novel continuous layer pruning framework that introduces two key innovations: a differentiable concave gate algorithm that automatically identifies the best continuous layer segments for pruning via gradient-based optimization; and a cutoff endpoint tuning strategy that effectively restores model performance by fine-tuning only the layers adjacent to the pruned segments. Extensive experiments across multiple model architectures (including LLaMA2, LLaMA3 and Qwen) and sizes (from $7$B to $70$B parameters) show that CLP significantly outperforms existing state-of-the-art baselines. For example, at a pruning rate of $20\%$, CLP achieves an average performance retention of $95.34\%$ on LLaMA3-70B, outperforming baselines by $4.29\%$-$30.52\%$. Furthermore, CLP can be seamlessly combined with quantization to further compress the model with only a slight performance loss.

Figures (6)

• Figure 1: Visualization of the differentiable concave gating algorithm.
• Figure 2: Visualization of pruned layers at $25\%$ pruning rate.
• Figure 3: Layer similarity visualization. The original LLaMA2-7B model is on the left, and the pruned model obtained by cutting off $25\%$ of the layers using our method is on the right.
• Figure 4: Visualization of the dynamic changes of $a$. The experiment is conducted on the LLaMA2-7B with the $25\%$ pruning rate.
• Figure 5: The actual speedup of pruned models obtained by our method.