LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

TL;DR

The paper tackles the computational burden of large language models by introducing LoRAShear, a structured pruning framework that leverages dependency graphs over LoRA adapters and a novel LHSPG optimizer to progressively prune while transferring knowledge to important structures. It adds a dynamic knowledge recovery stage that combines pretraining and instruction-tuning data to recover lost capabilities, enabling effective pruning under limited resources. Empirical results on LLAMAv1 show meaningful footprint reductions (20-50%) with minimal performance loss and outperformance of prior methods, highlighting practical viability for deploying compact LLMs. The work contributes a graph-based pruning paradigm, a specialized sparsity optimizer, and a dual-phase knowledge restoration process for robust model compression.

Abstract

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear.

Figures (4)

• Figure 1: Overview of LoRAShear. Given a general LLM, LoRAShear at first discovers the minimally removal structures, then analyzes the knowledge distribution to mark the crucial ones as unprunable, then performs progressive structurally pruning over the prunable structures via LHSPG, and finally recovers the lost knowledge to recap the performance gap to the full LLM.
• Figure 2: Dependency graph in LLAMAv1 and trainable variable partitions.
• Figure 3: Half-Space step over LoRA modules.
• Figure 4: Knowledge distribution analysis by measuring the perplexity deviation to the full LLAMAv1.