UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

TL;DR

UniComp provides a unified, capability-aware framework to evaluate LLM compression via pruning, quantization, and distillation across performance, reliability, and efficiency. By using 13 metrics on 40+ datasets and six compression techniques, it reveals a consistent knowledge bias where factual recall persists but reasoning, multilinguality, and instruction-following degrade, with quantization offering the best overall trade-offs and distillation delivering the strongest runtime gains at high compute cost. The study also shows that calibration data tailored to reasoning can substantially boost pruning-based reasoning without harming knowledge retention, while reliability does not perfectly track capability preservation. Practically, UniComp informs deployment choices by highlighting that performance and behavioral robustness are largely orthogonal under compression, underscoring the need for multi-dimensional evaluation in real-world settings.

Abstract

Model compression is increasingly essential for deploying large language models (LLMs), yet existing evaluations are limited in method coverage and focus primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through extensive evaluation of six compression techniques on modern LLMs across more than 40 datasets, we find that (i) compression exhibits a consistent knowledge bias, where knowledge-intensive tasks are relatively preserved while reasoning, multilingual, and instruction-following capabilities degrade substantially; (ii) quantization provides the best overall trade-off between retained performance and efficiency, whereas distillation yields strong runtime acceleration gains at high computational cost; and (iii) task-specific calibration can significantly improve the reasoning ability of pruned models by up to 50%.

Figures (5)

• Figure 1: Overview of our compression evaluation framework and results: (a) UniComp covers performance, reliability, and efficiency with 13 metrics; and (b) Knowledge bias in LLM compression. On LLaMA-3.1-8B, compression preserves knowledge performance but leads to pronounced degradation in multilingual and cultural, reasoning, and instruction following, with quantization as a partial exception.
• Figure 2: Efficiency results for compression of Qwen-2.5-7B in three dimensions: Runtime Acceleration $\mathcal{S}_{\text{RA}}$, Inference Efficiency $\mathcal{S}_{\text{IE}}$ and Compute Cost $\mathcal{S}_{\text{CC}}$.
• Figure 3: More analysis on compressed methods on different model comparison and effect of calibration: (a) six different reasoning model sizes; (b) three different model types; refer to Appendix \ref{['sec:ax_extended_tables']} for detailed results; and (c) calibration effects on two models, where reasoning-aware calibration strategy effectively improves reasoning performance without knowledge degradation.
• Figure 4: Knowledge vs. reasoning benchmark performance across compression techniques.
• Figure 5: Knowledge Bias in Qwen-2.5-7B model. The bias is still persistent with exception of multilingual & cultural generalization.