Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models

Changhai Zhou, Yuhua Zhou, Shijie Han, Qian Qiao, Hongguang Li

TL;DR

QPruner tackles the memory bottlenecks of large language models by coupling structured pruning with layer-wise mixed-precision quantization and PB-based precision optimization. The framework prioritizes per-layer importance, using mutual information to guide bit-width allocation and Bayesian optimization to refine configurations under memory budgets, followed by PEFT-based fine-tuning for performance recovery. Empirical results show substantial memory reductions with maintained or improved accuracy, especially at higher pruning rates, outperforming prior pruning-only baselines on several benchmarks. This approach enables more scalable deployment of LLMs in resource-constrained environments by balancing model size, speed, and accuracy.

Abstract

The rise of large language models (LLMs) has significantly advanced various natural language processing (NLP) tasks. However, the resource demands of these models pose substantial challenges. Structured pruning is an effective approach to reducing model size, but it often results in significant accuracy degradation, necessitating parameter updates to adapt. Unfortunately, such fine-tuning requires substantial memory, which limits its applicability. To address these challenges, we introduce quantization into the structured pruning framework to reduce memory consumption during both fine-tuning and inference. However, the combined errors from pruning and quantization increase the difficulty of fine-tuning, requiring a more refined quantization scheme. To this end, we propose QPruner, a novel framework that employs structured pruning to reduce model size, followed by a layer-wise mixed-precision quantization scheme. Quantization precisions are assigned to each layer based on their importance to the target task, and Bayesian optimization is employed to refine precision allocation strategies, ensuring a balance between model accuracy and memory efficiency. Extensive experiments on benchmark datasets demonstrate that QPruner significantly outperforms existing methods in memory savings while maintaining or improving model performance.

QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models

TL;DR

QPruner tackles the memory bottlenecks of large language models by coupling structured pruning with layer-wise mixed-precision quantization and PB-based precision optimization. The framework prioritizes per-layer importance, using mutual information to guide bit-width allocation and Bayesian optimization to refine configurations under memory budgets, followed by PEFT-based fine-tuning for performance recovery. Empirical results show substantial memory reductions with maintained or improved accuracy, especially at higher pruning rates, outperforming prior pruning-only baselines on several benchmarks. This approach enables more scalable deployment of LLMs in resource-constrained environments by balancing model size, speed, and accuracy.

Abstract

The rise of large language models (LLMs) has significantly advanced various natural language processing (NLP) tasks. However, the resource demands of these models pose substantial challenges. Structured pruning is an effective approach to reducing model size, but it often results in significant accuracy degradation, necessitating parameter updates to adapt. Unfortunately, such fine-tuning requires substantial memory, which limits its applicability. To address these challenges, we introduce quantization into the structured pruning framework to reduce memory consumption during both fine-tuning and inference. However, the combined errors from pruning and quantization increase the difficulty of fine-tuning, requiring a more refined quantization scheme. To this end, we propose QPruner, a novel framework that employs structured pruning to reduce model size, followed by a layer-wise mixed-precision quantization scheme. Quantization precisions are assigned to each layer based on their importance to the target task, and Bayesian optimization is employed to refine precision allocation strategies, ensuring a balance between model accuracy and memory efficiency. Extensive experiments on benchmark datasets demonstrate that QPruner significantly outperforms existing methods in memory savings while maintaining or improving model performance.

Paper Structure

This paper contains 22 sections, 10 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Quantization
  4. The Motivating Example
  5. QPruner
  6. Structured Pruning
  7. Mixed-Precision Quantization
  8. Performance Recovery
  9. Experiments
  10. Main Results
  11. Ablation Study
  12. Related Work
  13. Efficient Compression of LLMs
  14. Parameter Efficient Fine-Tuning
  15. Conclusion
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Comparison of accuracy and memory usage across different fine-tuning configurations for multiple tasks. The bars represent the accuracy of three different methods (LoRA, LoftQ, LoftQ*) on each task, while the markers indicate the memory usage for each corresponding method.
  • Figure 2: Overview of the QPruner framework.
  • Figure 3: Pareto-front scatter plots for BoolQ and WinoGrande with 50 data points. The red points indicate the non-dominated configurations within the Pareto frontier.
  • Figure 4: Pareto-front scatter plots for different Downstream Tasks