Model Compression and Efficient Inference for Large Language Models: A Survey

TL;DR

This survey addresses the growing challenge of deploying large language models by categorizing and evaluating algorithmic strategies for model compression and efficient inference. It systematically covers quantization, pruning, distillation, compact architecture design, and dynamic networks, emphasizing the distinct considerations for medium versus real large models, including finetuning cost and preservation of generalization. The authors discuss PTQ and QAT, weight- and activation-based strategies, and hardware-aware methods, while also detailing dynamic MoE architectures and inference acceleration frameworks. A key contribution is the synthesis of how these methods can be combined (e.g., MoE with quantization or pruning) to achieve practical, scalable deployment without sacrificing emergent capabilities. The work highlights open challenges and future directions, such as integrating pruning with quantization and distillation, improving training efficiency, and developing hardware-friendly kernels for truly scalable LLM inference.

Abstract

Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resource-constrained devices. In this paper, we investigate compression and efficient inference methods for large language models from an algorithmic perspective. Regarding taxonomy, similar to smaller models, compression and acceleration algorithms for large language models can still be categorized into quantization, pruning, distillation, compact architecture design, dynamic networks. However, Large language models have two prominent characteristics compared to smaller models: (1) Most of compression algorithms require finetuning or even retraining the model after compression. The most notable aspect of large models is the very high cost associated with model finetuning or training. Therefore, many algorithms for large models, such as quantization and pruning, start to explore tuning-free algorithms. (2) Large models emphasize versatility and generalization rather than performance on a single task. Hence, many algorithms, such as knowledge distillation, focus on how to preserving their versatility and generalization after compression. Since these two characteristics were not very pronounced in early large models, we further distinguish large language models into medium models and ``real'' large models. Additionally, we also provide an introduction to some mature frameworks for efficient inference of large models, which can support basic compression or acceleration algorithms, greatly facilitating model deployment for users.

Figures (8)

• Figure 1: The Transformer architecture drawn from MoE-Transformer.
• Figure 2: (a) Uniform quantization separates a real-valued range into uniform, finite intervals and then maps real values within the same interval to the same integer. (b) An FP16 tensor is quantized into INT4 format and then dequantized back into FP16.
• Figure 3: Three classes of static pruning methods. (a) Pre-training pruning; (b) During-training pruning; (c) Post-training pruning.
• Figure 4: The approximate probability density histogram of hard concrete distribution by using Monte Carlo simulation. The parameters of this hard concrete distribution are $\log{\alpha}=0$, $\beta=0.5$, $\gamma=-0.1$, and $\zeta=1.1$. Under this specification the hard concrete distribution assigns, roughly, half of its mass to {0, 1} and the rest to (0, 1).
• Figure 5: Taxonomy of knowledge distillation