Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PIP: Perturbation-based Iterative Pruning for Large Language Models

Yi Cao, Wei-Jie Xu, Yucheng Shen, Weijie Shi, Chi-Min Chan, Jianfeng Qu, Jiajie Xu

TL;DR

PIP tackles the deployment challenge of ultra-large language models by introducing a perturbation-based, double-view structured pruning framework. It combines unperturbed and perturbed input views and measures layer importance with a gradient-based PertImport metric, followed by an iterative pruning loop guarded by a consistency discriminator. Empirical results on LLaMA2/LLama3 demonstrate that roughly 20% pruning preserves over 85% of accuracy on diverse benchmarks and often matches dense-model performance within 5%, outperforming existing SOTA methods. The approach enables efficient, robust pruning that is complementary to quantization and can operate with very small calibration data, broadening practical applicability for edge and constrained environments.

Abstract

The rapid increase in the parameter counts of Large Language Models (LLMs), which often reach into the billions or even trillions, presents significant challenges for their practical deployment, particularly in resource-constrained environments. To address this issue, we propose PIP (Perturbation-based Iterative Pruning), a novel double-view structured pruning method to optimize LLMs, which combines information from two different views: the unperturbed view and the perturbed view. With the calculation of gradient differences, PIP iteratively prunes those that struggle to distinguish between these two views. Our experiments show that PIP reduces the parameter count by approximately 20% while retaining over 85% of the original model's accuracy across varied benchmarks. In some cases, the performance of the pruned model is within 5% of the unpruned version, demonstrating PIP's ability to preserve key aspects of model effectiveness. Moreover, PIP consistently outperforms existing state-of-the-art (SOTA) structured pruning methods, establishing it as a leading technique for optimizing LLMs in constrained environments.

PIP: Perturbation-based Iterative Pruning for Large Language Models

TL;DR

PIP tackles the deployment challenge of ultra-large language models by introducing a perturbation-based, double-view structured pruning framework. It combines unperturbed and perturbed input views and measures layer importance with a gradient-based PertImport metric, followed by an iterative pruning loop guarded by a consistency discriminator. Empirical results on LLaMA2/LLama3 demonstrate that roughly 20% pruning preserves over 85% of accuracy on diverse benchmarks and often matches dense-model performance within 5%, outperforming existing SOTA methods. The approach enables efficient, robust pruning that is complementary to quantization and can operate with very small calibration data, broadening practical applicability for edge and constrained environments.

Abstract

The rapid increase in the parameter counts of Large Language Models (LLMs), which often reach into the billions or even trillions, presents significant challenges for their practical deployment, particularly in resource-constrained environments. To address this issue, we propose PIP (Perturbation-based Iterative Pruning), a novel double-view structured pruning method to optimize LLMs, which combines information from two different views: the unperturbed view and the perturbed view. With the calculation of gradient differences, PIP iteratively prunes those that struggle to distinguish between these two views. Our experiments show that PIP reduces the parameter count by approximately 20% while retaining over 85% of the original model's accuracy across varied benchmarks. In some cases, the performance of the pruned model is within 5% of the unpruned version, demonstrating PIP's ability to preserve key aspects of model effectiveness. Moreover, PIP consistently outperforms existing state-of-the-art (SOTA) structured pruning methods, establishing it as a leading technique for optimizing LLMs in constrained environments.
Paper Structure (40 sections, 1 theorem, 15 equations, 5 figures, 15 tables, 1 algorithm)

This paper contains 40 sections, 1 theorem, 15 equations, 5 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Text Perturbation
  5. Character Swap
  6. Character Replacement
  7. Character Insertion
  8. PertImport: A Perturbation-based Metric for Layer Importance Assessment
  9. PIP: Perturbation-based Iterative Pruning
  10. Experiments
  11. Experimental Setup
  12. Model Selection
  13. Evaluation and Datasets
  14. Baseline Methods
  15. Experimental Details
  16. ...and 25 more sections

Key Result

Theorem 1

To enhance the robustness of the pruned model (defined as its capability to distinguish between $s$ and $s+\delta s$), it's best to select parameters with smaller gradient differences between the perturbed and unperturbed views.

Figures (5)

  • Figure 1: Generation of perturbed texts via auxiliary LLM.
  • Figure 2: Overview of our PIP method, where an auxiliary LLM generates perturbed text (see Section \ref{['subsection:perturbation']}). During pruning, the same original and perturbed texts are used to compute PertImport values, determining layer significance. The least significant layers (e.g., the 2nd, ($N-1$)-th, and $N$-th layers in the first three steps) are iteratively removed.
  • Figure 3: Zero-shot performance of the pruned LLM using PIP, across various gradient aggregation strategies.
  • Figure 4: Experiments of performance at various data sizes. "Data Size" is the number of samples in the calibration dataset. "Relative Accuracy" is the ratio of the average accuracy of the pruned LLM on various benchmarks to the average accuracy of the Dense model.
  • Figure 5: Impact of pruning ratio on pruning time.

Theorems & Definitions (3)

  • Theorem 1
  • proof
  • proof