Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compressing LLMs with MoP: Mixture of Pruners

Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Leandro Giusti Mugnaini, Keith Ando Ogawa, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

TL;DR

MoP addresses the high compute cost of large language models by introducing Mixture of Pruners, a framework that unifies depth and width pruning. It iteratively creates two pruned candidates per step—one by removing a transformer layer and one by pruning internal widths—and uses a path criterion to select the next step, with a brief recovery fine-tuning after each candidate. Key design choices include pruning the third-to-last layer as the depth criterion and using AMP-based width pruning to match parameter budgets, with path metrics such as cosine similarity, KL divergence, or perplexity (or random) guiding selection. Empirically, MoP achieves state-of-the-art accuracy across 20%, 30%, and 40% compression on LLaMA-2 7B and LLaMA-3 8B and delivers substantial speedups (up to about 39% end-to-end latency reduction at 40% compression) and robust multimodal transfer when applied to LLaVA-1.5, demonstrating a hardware-friendly, versatile pruning paradigm.

Abstract

The high computational demands of Large Language Models (LLMs) motivate methods that reduce parameter count and accelerate inference. In response, model pruning emerges as an effective strategy, yet current methods typically focus on a single dimension-depth or width. We introduce MoP (Mixture of Pruners), an iterative framework that unifies these dimensions. At each iteration, MoP generates two branches-pruning in depth versus pruning in width-and selects a candidate to advance the path. On LLaMA-2 and LLaMA-3, MoP advances the frontier of structured pruning, exceeding the accuracy of competing methods across a broad set of compression regimes. It also consistently outperforms depth-only and width-only pruning. Furthermore, MoP translates structural pruning into real speedup, reducing end-to-end latency by 39% at 40% compression. Finally, extending MoP to the vision-language model LLaVA-1.5, we notably improve computational efficiency and demonstrate that text-only recovery fine-tuning can restore performance even on visual tasks.

Compressing LLMs with MoP: Mixture of Pruners

TL;DR

MoP addresses the high compute cost of large language models by introducing Mixture of Pruners, a framework that unifies depth and width pruning. It iteratively creates two pruned candidates per step—one by removing a transformer layer and one by pruning internal widths—and uses a path criterion to select the next step, with a brief recovery fine-tuning after each candidate. Key design choices include pruning the third-to-last layer as the depth criterion and using AMP-based width pruning to match parameter budgets, with path metrics such as cosine similarity, KL divergence, or perplexity (or random) guiding selection. Empirically, MoP achieves state-of-the-art accuracy across 20%, 30%, and 40% compression on LLaMA-2 7B and LLaMA-3 8B and delivers substantial speedups (up to about 39% end-to-end latency reduction at 40% compression) and robust multimodal transfer when applied to LLaVA-1.5, demonstrating a hardware-friendly, versatile pruning paradigm.

Abstract

The high computational demands of Large Language Models (LLMs) motivate methods that reduce parameter count and accelerate inference. In response, model pruning emerges as an effective strategy, yet current methods typically focus on a single dimension-depth or width. We introduce MoP (Mixture of Pruners), an iterative framework that unifies these dimensions. At each iteration, MoP generates two branches-pruning in depth versus pruning in width-and selects a candidate to advance the path. On LLaMA-2 and LLaMA-3, MoP advances the frontier of structured pruning, exceeding the accuracy of competing methods across a broad set of compression regimes. It also consistently outperforms depth-only and width-only pruning. Furthermore, MoP translates structural pruning into real speedup, reducing end-to-end latency by 39% at 40% compression. Finally, extending MoP to the vision-language model LLaVA-1.5, we notably improve computational efficiency and demonstrate that text-only recovery fine-tuning can restore performance even on visual tasks.
Paper Structure (14 sections, 3 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 14 sections, 3 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Mixture of Pruners (MoP) method
  4. Transformer Architecture
  5. Proposed method
  6. Path Selection Strategies
  7. Experiments
  8. Experimental Settings
  9. Comparison of Path Selection Strategies.
  10. Zero-Shot Performance
  11. Inference Speedup
  12. MoP on Multimodal Architectures
  13. Conclusions
  14. Acknowledgments

Figures (3)

  • Figure 1: Illustration of MoP combining depth pruning (removing transformer layers) and width pruning (removing attention heads and MLP neurons).
  • Figure 2: Average accuracy on five standard commonsense benchmarks, comparing depth pruning, width pruning, and MoP (combining both) on LLaMA-2 7B. MoP achieves a Pareto improvement over single-dimension approaches.
  • Figure 3: Impact of pruning on LLaVA-1.5 7B with MoP and recovery with text-only fine-tuning. We present the mean accuracy of three runs for ScienceQA and VizWiz across varying compression ratios. Shaded regions denote the standard deviation.