Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation

TL;DR

This paper tackles the challenge of compressing large language models without sacrificing performance. It challenges the effectiveness of direct layer pruning and linear weight aggregation, and introduces CoMe, which combines a channel sensitivity metric, progressive concatenation-based layer merging, and a hierarchical distillation post-training protocol. Across seven models and multiple sparsities, CoMe achieves state-of-the-art results, with significant performance retention (e.g., 83% Avg retention at 30% pruning for LLaMA-2-7b) and superior post-training recovery, especially when using the CoMe-sp single-process distillation. The work advances practical, hardware-friendly model compression and offers a pathway to deploy capable, efficient LLMs in resource-constrained settings through principled channel-level preservation and coordinated knowledge transfer.

Abstract

Large Language Models excel at natural language processing tasks, but their massive size leads to high computational and storage demands. Recent works have sought to reduce their model size through layer-wise structured pruning. However, they tend to ignore retaining the capabilities in the pruned part. In this work, we re-examine structured pruning paradigms and uncover several key limitations: 1) notable performance degradation due to direct layer removal, 2) incompetent linear weight layer aggregation, and 3) the lack of effective post-training recovery mechanisms. To address these limitations, we propose CoMe, including a progressive layer pruning framework with a Concatenation-based Merging technology and a hierarchical distillation post-training process. Specifically, we introduce a channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection. Subsequently, we employ a concatenation-based layer merging method to fuse the most critical channels across adjacent layers, enabling progressive model size reduction. Finally, we propose a hierarchical distillation protocol that leverages the correspondences between the original and pruned model layers established during pruning, thereby enabling efficient knowledge transfer. Experiments on seven benchmarks show that CoMe achieves state-of-the-art performance; when pruning 30% of LLaMA-2-7b's parameters, the pruned model retains 83% of its original average accuracy. Our code is available at https://github.com/MPI-Lab/CoMe.

Figures (14)

• Figure 1: Overview of CoMe. Steps 1 and 2 are iteratively executed until the model achieves the target size. Following this, Step 3 utilizes efficient post-training through feature-level distillation to recover the performance. The resulting lightweight model incorporates several newly replaced layers.
• Figure 2: Comparative analysis of normalized layer importance score across different methods. The score of SLEB and FuseGPT is derived from the first-round pruning.
• Figure 3: Merge adjacent layers with linear weight aggregation. First, we reduce LLaMA-2-7b's layer number to 23 with ShortGPT. Second, the components in layers 19 and 20 are merged at different ratios. When merging MHA or FFN, the other component is retained from layer 20.
• Figure 4: Channel importance calculation process. The importance scores reflect the expected changes of output caused by different channels.
• Figure 5: Concatenation-based weight merge process. $W^{(l)}$ and $W^{(l+1)}$ are derived from the same positions in adjacent layers.