Semantic Retention and Extreme Compression in LLMs: Can We Have Both?

TL;DR

This work presents a theoretical and empirical study of joint pruning and quantization for large language models. It introduces the Theoretical Compression Rate ($TCr$) to compare configurations independent of hardware overheads, and defines semantic retention-based metrics, notably the Semantic Retention Rate ($Sr$) and Semantic Retention Compression Rate ($SrCr$), to quantify the trade-off between size and capability. Through experiments on open-source models (e.g., Llama-3.1-8B and Mistral-7B) and robust benchmarks (MMLU-Pro, BBH, MATH), the authors show that well-balanced joint configurations (such as 25% pruning with 4-bit quantization) can outperform single-method baselines by roughly 20% in semantic retention at the same $TCr$, and that semi-structured pruning patterns offer hardware-efficient paths forward. The study highlights that joint optimization, informed by the $SrCr$ framework and hardware considerations, can enable more practical and scalable deployment of LLMs in resource-constrained environments. Limitations include reliance on sequential approximations and evaluations on smaller models, pointing to future work in unified joint optimization and hardware-aware, dynamic compression methods.

Abstract

The exponential growth in Large Language Model (LLM) deployment has intensified the need for efficient model compression techniques to reduce computational and memory costs. While pruning and quantization have shown promise, their combined potential remains largely unexplored. In this paper, we examine joint compression and how strategically combining pruning and quantization could yield superior performance-to-compression ratios compared to single-method approaches. Recognizing the challenges in accurately assessing LLM performance, we address key limitations of previous evaluation frameworks and introduce the Semantic Retention Compression Rate (SrCr), a novel metric that quantifies the trade-off between model compression and semantic preservation, facilitating the optimization of pruning-quantization configurations. Experiments demonstrate that our recommended combination achieves, on average, a 20% performance increase compared to an equivalent quantization-only model at the same theoretical compression rate.

Figures (11)

• Figure 1: Retention rates ($R_t$ across tasks and $Sr$) with pruning-only at different sparsity levels (25%, 33.333%, 50%, and 75%).
• Figure 2: Retention rates ($R_t$ across tasks and $Sr$) with quantization-only at different bit-widths (8, 4, 3, and 2).
• Figure 3: Semantic retention compression rates for single compression methods.
• Figure 4: Error $\|\delta_{A,j}\|_2^2$ analysis by comparison with NF4 and LLM.int8().
• Figure 5: Retention rates ($R_t$ across tasks and $Sr$) for joint compression vs. quantization-only at different $TCr$.