Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization

TL;DR

This work proposes Quant Experts (QE), a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization, which consistently enhances task accuracy across various quantization settings and model scales, while maintaining performance comparable to full-precision models.

Abstract

Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and memory overheads of Vision-Language Models (VLMs) by compressing both weights and activations without retraining the full model. Existing PTQ methods primarily rely on static identification and global compensation of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Accordingly, we propose \textbf{Quant Experts (QE)}, a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization. QE divides the important channels into token-independent and token-dependent groups. For the former, a shared expert is designed for most tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error related to specific tokens. Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B parameters, while maintaining performance comparable to full-precision models.

Figures (6)

• Figure 1: Illustration of Different Quantization Granularities. $\mathcal{D}$ is the calibration data, and $\mathbf{W}_q$ is the quantized weight matrix. SmoothQuant smoothquant employs channel-wise with fixed scaling coefficients to achieve global quantization. MBQ Mbq presents a modality-aware strategy that focuses on sensitivities in input modalities. In contrast, we propose a token-aware adaptive quantization that considers both global and local error reconstruction.
• Figure 2: Visualization of a Transformer block in Qwen2VL-2B, illustrating token value distributions (top) and the positions of important channels (bottom). In the top panel, brightness reflects token magnitude, while in the bottom panel, highlighted positions denote important channels.
• Figure 3: Visualization of a Transformer block in Qwen2VL-2B, illustrating important channel behavior. The top panel shows activation frequencies, while the bottom depicts average activation values. Red crosses mark important channel positions identified by the static global method on the calibration dataset.
• Figure 4: The framework of Quant Experts (QE). The token-independent channels are model by a shared expert, while token-dependent channels are captured by multiple routed experts. A lightweight low-rank adapter is implemented for each expert.
• Figure 5: Illustration of the Inference Computation Process of QE.