SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment

TL;DR

SLMQuant addresses a critical gap in deploying small language models by benchmarking how LLM-oriented quantization methods perform on SLMs. It introduces a three-track framework—compression performance, inference performance, and quantization consumption—to systematically compare SmoothQuant, OmniQuant, and SpinQuant across multiple SLM architectures and tasks, revealing fundamental bottlenecks distinct from LLMs. The findings show that 8-bit quantization is generally safe for SLMs, while low-bit quantization can severely degrade performance, and that architecture-specific features like NoPE, YaRN, and GQA influence quantization resilience. These insights lay the groundwork for SLM-tailored compression design and practical edge deployment, with practical guidance and a foundation for future expansion of SLM quantization research.

Abstract

Despite the growing interest in Small Language Models (SLMs) as resource-efficient alternatives to Large Language Models (LLMs), their deployment on edge devices remains challenging due to unresolved efficiency gaps in model compression. While quantization has proven effective for LLMs, its applicability to SLMs is significantly underexplored, with critical questions about differing quantization bottlenecks and efficiency profiles. This paper introduces SLMQuant, the first systematic benchmark for evaluating LLM compression techniques when applied to SLMs. Through comprehensive multi-track evaluations across diverse architectures and tasks, we analyze how state-of-the-art quantization methods perform on SLMs. Our findings reveal fundamental disparities between SLMs and LLMs in quantization sensitivity, demonstrating that direct transfer of LLM-optimized techniques leads to suboptimal results due to SLMs' unique architectural characteristics and training dynamics. We identify key factors governing effective SLM quantization and propose actionable design principles for SLM-tailored compression. SLMQuant establishes a foundational framework for advancing efficient SLM deployment on low-end devices in edge applications, and provides critical insights for deploying lightweight language models in resource-constrained scenarios.

Figures (4)

• Figure 1: The motivation of SLMQuant.
• Figure 2: Overview of our SLMQuant, which consists of three evaluation tracks: compression performance, inference consumption, and quantization consumption.
• Figure 3: The weight distributions of LLM and SLM.
• Figure 4: Overall comparison of different quantization methods in our SLMQuant.