LLMs can Compress LLMs: Adaptive Pruning by Agents

TL;DR

The paper tackles the problem of pruning large language models without retraining, addressing the knowledge degradation seen with existing structured pruning. It introduces an adaptive pruning framework in which a foundation model acts as an autonomous pruning agent, guided by layer-wise sensitivity profiling that combines Wanda-like metrics with gradient importance, all normalized via z-scores. A self-reflection loop and a checkpoint rollback mechanism enable the agent to learn from pruning outcomes and avoid detrimental degradation, achieving substantial gains over structured baselines on Qwen3 models (~45-50% sparsity). The approach demonstrates strong improvements in MMLU accuracy and factual knowledge retention (e.g., FreebaseQA) while keeping perplexity degradation low, suggesting that foundation models can effectively guide the compression of other foundation models without retraining and with robust self-correction.

Abstract

As Large Language Models (LLMs) continue to scale, post-training pruning has emerged as a promising approach to reduce computational costs while preserving performance. Existing methods such as SparseGPT and Wanda achieve high sparsity through layer-wise weight reconstruction or activation-aware magnitude pruning, but rely on uniform or hand-crafted heuristics to determine per-layer sparsity ratios. Moreover, recent work has shown that pruned LLMs suffer from severe factual knowledge degradation, with structured pruning methods experiencing near-total collapse in factual question-answering capabilities. We introduce agent-guided pruning, where a foundation model acts as an adaptive pruning agent to intelligently select which layers to prune at each iteration while preserving critical knowledge pathways. Our method constructs layer-wise sensitivity profiles by combining Wanda-inspired weight-activation metrics with gradient importance scores, normalized as z-scores for model-agnostic comparison. These statistics are processed by an LLM agent equipped with self-reflection capabilities, enabling it to learn from previous pruning outcomes and iteratively refine its strategy. A checkpoint rollback mechanism maintains model quality by reverting when perplexity degradation exceeds a threshold. We evaluate our approach on Qwen3 models (4B and 8B parameters) at approximately 45% sparsity, demonstrating substantial improvements over structured pruning baselines: 56% relative improvement in MMLU accuracy, 19x better factual knowledge retention on FreebaseQA, and 69% lower perplexity degradation. Notably, our framework requires no retraining, operates in a model-agnostic manner, and exhibits effective self-correction with only 2-4 rollbacks across 21-40 iterations, demonstrating that foundation models can effectively guide the compression of other foundation models.

Figures (6)

• Figure 1: Cumulative statistics for agent-guided pruning on Qwen3-8B across 21 iterations. Top left: Sparsity evolution showing gradual progression to 50% target. Top right: Perplexity evolution showing controlled degradation with 2 rollback events. Bottom left: Per-iteration sparsity gains, with most iterations achieving 1-3% progress. Bottom right: Per-iteration perplexity changes, showing the agent learns to keep PPL increases below 2% in most iterations.
• Figure 2: Cumulative statistics for agent-guided pruning on Qwen3-4B across 40 iterations. The agent achieves larger sparsity gains (3-9%) in early iterations when the model is robust, then becomes more conservative as perplexity rises. Four rollback events (iterations 10, 15, 25, 32) are followed by visible strategy adjustments, demonstrating effective learning from negative feedback.
• Figure 3: Performance comparison on Qwen3-8B at $\sim$43% sparsity across three evaluation metrics. Our agent-guided method substantially outperforms structured pruning baselines, particularly in preserving factual knowledge (FreebaseQA) where structured methods experience catastrophic degradation.
• Figure 4: Performance comparison on Qwen3-4B at $\sim$45% sparsity. The pattern of improvements is consistent with the 8B model, demonstrating that agent-guided pruning generalizes effectively across model scales.
• Figure 5: MMLU performance by category for Qwen3-8B. Our method maintains substantially better performance than structured baselines across all knowledge domains, with Social Sciences showing the strongest retention.