Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPAP: Structured Pruning via Alternating Optimization and Penalty Methods

Hanyu Hu, Xiaoming Yuan

TL;DR

SPAP addresses the practical deployment challenge of large language models by introducing a structured pruning framework grounded in optimization. It formulates pruning of GLU-based MLP blocks as a mixed-integer problem and proves a continuous relaxation preserves optimality (Theorem 1), then solves it with a penalty method that enforces sparsity via a parameter $\rho$ and a carefully designed $\bm{s}$-update along with a closed-form $\bm{W}$-update $\bm{W} = \bm{Y}\bm{X}^\top (\bm{X}\bm{X}^\top + \rho \mathrm{diag}(\bm{s}))^{-1}$. An alternating minimization strategy updates the up, gate, and down projections by combining gradient steps for $\bm{W}_{up}$ and $\bm{W}_{gate}$ with a closed-form $\bm{W}_{down}$ update using $\bm{Z} = \bm{W}_{up}\bm{X} \odot \sigma(\bm{W}_{gate}\bm{X})$. Empirical results on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 show SPAP delivers near-linear speedups (e.g., about $1.29\times$ at $30\%$ sparsity) and proportional memory reductions with strong perplexity and zero-shot performance, demonstrating an effective, optimization-driven path to hardware-agnostic LLM pruning.

Abstract

The deployment of large language models (LLMs) is often constrained by their substantial computational and memory demands. While structured pruning presents a viable approach by eliminating entire network components, existing methods suffer from performance degradation, reliance on heuristic metrics, or expensive finetuning. To address these challenges, we propose SPAP (Structured Pruning via Alternating Optimization and Penalty Methods), a novel and efficient structured pruning framework for LLMs grounded in optimization theory. SPAP formulates the pruning problem through a mixed-integer optimization model, employs a penalty method that effectively makes pruning decisions to minimize pruning errors, and introduces an alternating minimization algorithm tailored to the splittable problem structure for efficient weight updates and performance recovery. Extensive experiments on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models demonstrate SPAP's superiority over state-of-the-art methods, delivering linear inference speedups (1.29$\times$ at 30% sparsity) and proportional memory reductions. Our work offers a practical, optimization-driven solution for pruning LLMs while preserving model performance.

SPAP: Structured Pruning via Alternating Optimization and Penalty Methods

TL;DR

SPAP addresses the practical deployment challenge of large language models by introducing a structured pruning framework grounded in optimization. It formulates pruning of GLU-based MLP blocks as a mixed-integer problem and proves a continuous relaxation preserves optimality (Theorem 1), then solves it with a penalty method that enforces sparsity via a parameter and a carefully designed -update along with a closed-form -update . An alternating minimization strategy updates the up, gate, and down projections by combining gradient steps for and with a closed-form update using . Empirical results on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 show SPAP delivers near-linear speedups (e.g., about at sparsity) and proportional memory reductions with strong perplexity and zero-shot performance, demonstrating an effective, optimization-driven path to hardware-agnostic LLM pruning.

Abstract

The deployment of large language models (LLMs) is often constrained by their substantial computational and memory demands. While structured pruning presents a viable approach by eliminating entire network components, existing methods suffer from performance degradation, reliance on heuristic metrics, or expensive finetuning. To address these challenges, we propose SPAP (Structured Pruning via Alternating Optimization and Penalty Methods), a novel and efficient structured pruning framework for LLMs grounded in optimization theory. SPAP formulates the pruning problem through a mixed-integer optimization model, employs a penalty method that effectively makes pruning decisions to minimize pruning errors, and introduces an alternating minimization algorithm tailored to the splittable problem structure for efficient weight updates and performance recovery. Extensive experiments on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models demonstrate SPAP's superiority over state-of-the-art methods, delivering linear inference speedups (1.29 at 30% sparsity) and proportional memory reductions. Our work offers a practical, optimization-driven solution for pruning LLMs while preserving model performance.
Paper Structure (13 sections, 1 theorem, 20 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 13 sections, 1 theorem, 20 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Optimization Model for Structured Pruning
  5. A Penalty Method
  6. Updating the Up and Gate Projections
  7. Experiments
  8. Experimental Setup
  9. Main Results
  10. Ablation Study
  11. Conclusion
  12. Proof of Theorem \ref{['thm:structured_relaxation']}
  13. Derivations of the W-subproblem in the Penalty Method

Key Result

Theorem 1

Consider the following relaxed optimization problem: For any optimal solution $(\hat{\bm{W}}, \hat{\bm{s}})$ of (eq:structured-relaxed), let $J$ be an arbitrary $\lambda$-subset of the support of $\hat{\bm{s}}$. Then, there exists another optimal solution $(\bm{W}', \bm{s}')$ to (eq:structured-relaxed) such that $\bm{s}' \in \{0,1\}^n$ and $\operatorn

Figures (2)

  • Figure 1: Overview of the SPAP method. Left: The layer-wise pruning problem of an MLP layer. Middle: We propose a penalty method to decide which parts of the weight matrices should be pruned. Right: We develop an alternating minimization algorithm to efficiently update the remaining weights to restore model performance.
  • Figure 2: Perplexity results of pruned LLaMA-3.2-1B and Qwen2.5-0.5B models under various sparsity. SPAP achieves significantly lower ppl in all settings than the baseline methods.

Theorems & Definitions (2)

  • Theorem 1
  • proof