The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM

TL;DR

This work presents a principled and effective method, called Elsa, which achieves extreme sparsity levels of up to 90% while retaining high model fidelity through standard and well-established constrained optimization techniques based on ADMM.

Abstract

Neural network pruning is a promising technique to mitigate the excessive computational and memory requirements of large language models (LLMs). Despite its promise, however, progress in this area has diminished, as conventional methods are seemingly unable to surpass moderate sparsity levels (50-60%) without severely degrading model accuracy. This work breaks through the current impasse, presenting a principled and effective method called $\texttt{Elsa}$, which achieves extreme sparsity levels of up to 90% while retaining high model fidelity. This is done by identifying several limitations in current practice, all of which can be traced back to their reliance on a surrogate objective formulation. $\texttt{Elsa}$ tackles this issue directly and effectively via standard and well-established constrained optimization techniques based on ADMM. Our extensive experiments across a wide range of models and scales show that $\texttt{Elsa}$ achieves substantial improvements over existing methods; e.g., it achieves 7.8$\times$ less perplexity than the best existing method on LLaMA-2-7B at 90% sparsity. Furthermore, we present $\texttt{Elsa}_{\text{-L}}$, a quantized variant that scales to extremely large models (27B), and establish its theoretical convergence guarantees. These results highlight meaningful progress in advancing the frontier of LLM sparsity, while promising that significant opportunities for further advancement may remain in directions that have so far attracted limited exploration.

Figures (6)

• Figure 1: Perplexity ($\downarrow$) vs. Sparsity ($\uparrow$) curves for different pruning methods; it is measured on the C4 dataset for pruned LLaMA-2-7B models. While existing methods start to fail as sparsity increases, our approach (Elsa) stays stable without losing much performance, revealing the unseen frontier. Previously it was considered nearly impossible to achieve such high sparsity for LLMs or go beyond the "sparsity wall" formed around 50-60% sparsity levels. The same trend is observed consistently across different architectures and scales as we will show in \ref{['sec:experiments']}--Experiments.
• Figure 2: Perplexity vs. Sparsity plots for different models and scales. Elsa preserves much lower perplexity at high sparsity compared to other methods, consistently across a wide range of settings, showing its advantage and robustness. All numerical results are provided in \ref{['Appendix:Additional results']}.
• Figure 3: Pareto optimality of Elsa compared to prior works in terms of perplexity vs. number of non-zero parameters. Elsa displays its greater optimality across a broad spectrum of effective scales.
• Figure 4: Zero-shot accuracy of pruned LLaMA-2-7B models. Elsa outperforms other methods for most tasks, with the performance gap widening as sparsity increases, highlighting its strong generalization capability. Full numerical results are provided in \ref{['tab:llama7b-zeroshot']} of Appendix \ref{['Appendix:Additional results']}.
• Figure 5: Perplexity of Gemma-2-27B. Elsa$_{\text{-L}}$ achieves the lowest perplexity, confirming its strength.

Theorems & Definitions (10)

• Definition 4.4
• Corollary 4.5
• Theorem 4.6
• Lemma A.1
• Lemma A.2
• proof
• Lemma A.3
• proof
• Theorem A.4
• proof