Beyond One-Size-Fits-All Pruning via Evolutionary Metric Search for Large Language Models

TL;DR

This work addresses the fragility of fixed pruning metrics for large language models by introducing OptiShear, a global evolutionary framework that automatically discovers adaptive pruning metrics tailored to specific model-task pairs. It defines a meta pruning metric that adaptively balances weights and activations via normalization and nonlinear transforms, coupled with a global divergence objective $\mathcal{L}_{\text{div}}$ and an NSGA-II search to optimize metric configurations. Through extensive experiments on 16 LLMs (across OPT, Mistral, LLaMA-1/2/3) and 10 tasks, OptiShear achieves superior compression performance without weight updates, demonstrates cross-task generalization and cross-model transferability of learned metrics, and markedly improves weight-activation alignment. The results suggest that adaptive, activation-aware pruning policies can provide robust, cost-effective model compression with practical deployment benefits, while highlighting opportunities for further universal metric design and improved structured pruning for highly knowledge-dense models.

Abstract

Post-training pruning has emerged as a crucial optimization technique as large language models (LLMs) continue to grow rapidly. However, the significant variations in weight distributions across different LLMs make fixed pruning strategies inadequate for multiple models. In this paper, we introduce \textbf{\textsc{OptiShear}}, an efficient evolutionary optimization framework for adaptive LLM pruning. Our framework features two key innovations: an effective search space built on our Meta pruning metric to handle diverse weight distributions, and a model-wise reconstruction error for rapid evaluation during search trials. We employ Non-dominated Sorting Genetic Algorithm III (NSGA-III) to optimize both pruning metrics and layerwise sparsity ratios. Through extensive evaluation on LLaMA-1/2/3 and Mistral models (7B-70B) across multiple benchmarks, we demonstrate that our adaptive pruning metrics consistently outperform existing methods. Additionally, our discovered layerwise sparsity ratios enhance the effectiveness of other pruning metrics. The framework exhibits strong cross-task and cross-model generalizability, providing a cost-effective solution for model compression.

Figures (10)

• Figure 1: Existing pruning metrics, which follow a one-size-fits-all design, fail on LLaMA-3 due to its distinct, highly concentrated weight distribution—marked by a symmetric bell-shaped curve with few near-zero weights—compared to the more dispersed patterns in LLaMA-1/2 and Mistral.
• Figure 2: WikiText perplexity of metrics transferred from GSM8K (mathematical reasoning) to language modeling, showing strong cross-task generalization.
• Figure 3: WikiText perplexity of metrics transferred from LLaMA-2 and LLaMA-3 to LLaMA-1 and Mistral models, showing strong cross-model transferability.
• Figure 4: Comparison of pruning methods on Mistral 7B.
• Figure 5: Sensitivity evaluation across sparsity levels, calibration data size, and reasoning steps in calibration samples for mathematical reasoning.