HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning

TL;DR

This work tackles the instability of post-training quantization (PTQ) for large language models by linking quantization sensitivity to the Hessian geometry of the loss landscape. It introduces HeRo-Q, a lightweight, architecture-agnostic preconditioning transform that combines diagonal smoothing $\mathbf{D}_{\alpha}$ and an orthogonal rotation $\mathbf{R}$ to form $\mathbf{T} = \mathbf{D}_{\alpha}^{-1}\mathbf{R}$, reducing the Hessian's spectral radius and rebalancing quantization noise. The authors provide theoretical justification via a spectral error bound and demonstrate empirically that HeRo-Q outperforms GPTQ, AWQ, SpinQuant, SmoothQuant, and OmniQuant across Llama and Qwen models, especially in ultra low-bit regimes like W3A16, with negligible inference overhead due to fusion and efficient online rotation. The approach enables reliable, high-fidelity quantization suitable for deploying large language models on more affordable hardware, with meaningful implications for energy efficiency and accessibility.

Abstract

Post Training Quantization (PTQ), a mainstream model compression technique, often leads to the paradoxical 'low error, high loss' phenomenon because it focuses solely on minimizing quantization error. The root cause lies in the Hessian matrix of the LLM loss landscape: a few high curvature directions are extremely sensitive to perturbations. To address this, we propose the Hessian Robust Quantization (HeRo Q) algorithm, which applies a lightweight, learnable rotation-compression matrix to the weight space prior to quantization. This joint framework reshapes the loss landscape by reducing the largest Hessian eigenvalue and reducing its max eigenvalue, thereby significantly enhancing robustness to quantization noise. HeRo-Q requires no architectural modifications, incurs negligible computational overhead, and integrates seamlessly into existing PTQ pipelines. Experiments on Llama and Qwen models show that HeRo Q consistently outperforms state of the art methods including GPTQ, AWQ, and SpinQuant not only achieving superior performance under standard W4A8 settings, but also excelling in the highly challenging W3A16 ultra low bit regime, where it boosts GSM8K accuracy on Llama3 8B to 70.15\% and effectively avoids the logical collapse commonly seen in aggressive quantization.

Figures (6)

• Figure 1: a: Certain parameters suffer severe performance degradation even under small quantization noise because their noise components project significantly onto the high-curvature (short-axis) directions. Noise amplitude is fixed across all trials in the figure. b: The local geometry of the loss landscape around a converged point can be approximated by a hyper-ellipsoid: its short axis corresponds to the direction of the largest Hessian eigenvalue (high curvature). In contrast, its long axis aligns with the smallest eigenvalue (low curvature). A small perturbation along the short axis causes a dramatic increase in loss, whereas a perturbation of the same magnitude along the long axis has a negligible effect.
• Figure 2: HeRo-Q stabilizes quantization by reshaping the Hessian spectrum. Through a lightweight linear transformation, HeRo-Q attenuates large eigenvalues—particularly outliers far from the bulk of the spectrum—and caps the maximum eigenvalue, thereby reducing the loss landscape’s sensitivity to quantization noise and enabling robust, efficient low-bit inference.
• Figure 3: Robust Quantization via Hessian Transformation: HeRo-Q first analyzes the Hessian structure and identifies a severe Hessian geometric skewness —certain directions exhibit extremely high curvature (corresponding to large eigenvalues) and are highly sensitive to quantization noise. To address this, HeRo-Q searches over smoothing parameters $\alpha$ to construct a diagonal scaling matrix $\mathbf{D}_\alpha$, and learns an orthogonal rotation matrix $\mathbf{R}$ via the Cayley transform. Together, they form a linear transformation $\mathbf{T} = \mathbf{D}_\alpha^{-1} \mathbf{R}$ that maps the original weight space into a geometrically more balanced transformed space. Robust quantization is then performed in this transformed space, and the quantized weights in the original space are recovered through the inverse transformation.
• Figure 4: Global Hessian Eigenspectrum Transformation. Comparison of the log-magnitude density of Hessian eigenvalues before (Red) and after (Green) applying HeRo-Q on Llama-1B (a) and Llama-3B (b).
• Figure 5: Layer-wise Spectral Radius ($\lambda_{max}$) Comparison. The original models (Red line) exhibit extreme volatility in spectral radius across layers, particularly in the embedding and output bottlenecks. HeRo-Q (Green line) acts as a spectral equalizer, consistently suppressing the peak curvature across all layers and reducing the variance of quantization difficulty.

Theorems & Definitions (6)

• Theorem 3.1: Spectral Surrogate Bound
• Theorem 3.2: Strict Reduction of Spectral Error Bound
• Theorem 1.1: Spectral Error Bound
• Theorem 1.2: Spectral Radius Upper Bound
• proof
• Theorem 1.3: Spectral Radius Compression Theorem