DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization

TL;DR

DartQuant tackles activation outliers in large language model quantization by reframing rotation optimization as distribution calibration. It introduces Whip loss to push rotated activations toward a near-uniform distribution and uses QR-Orth to enforce orthogonality efficiently, avoiding expensive manifold optimizers. The approach yields substantial practical benefits, achieving up to 47x speedups and 10x memory savings for rotation calibration on 70B models, and enabling calibration on a single 3090 GPU in about 3 hours. Empirical results across Llama-2/3 and MoE models demonstrate robust, state-of-the-art performance in 4-bit quantization settings, with strong cross-dataset stability and a clear potential to broaden LLM deployment in resource-constrained environments.

Abstract

Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47$\times$ acceleration and 10$\times$ memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments. Code is available at https://github.com/CAS-CLab/DartQuant.git.

Figures (13)

• Figure 1: Comparison of computational costs across different rotation optimization methods.
• Figure 2: Effects of different transformations on activation distribution.
• Figure 3: Effects of different transformations on 1000 activations in layer 20 for various models. The rotation matrix optimized by DartQuant achieves the lowest number of outliers and the smallest quantization error.
• Figure 4: Left: The DartQuant implementation process, with $Z$ representing the latent parameters in QR-orth and $R$ as the applied rotation matrix. Right: The change in rotation matrix before and after calibration.
• Figure 5: Intuition behind the distribution transformation: $U_X(x)$ transforms the Laplace distribution into a uniform distribution by flattening the peak and aggregating the outliers.