Boost Post-Training Quantization via Null Space Optimization for Large Language Models
Jiaqi Zhao, Miao Zhang, Deng Xiang, Ming Li, Weili Guan, Liqiang Nie
TL;DR
This work addresses the inevitable quantization error in post-training quantization of large language models by introducing a null-space optimization perspective. It proposes Q2N, a plug-in module that projects quantization perturbations into the input-activation null space to minimize impact on outputs, and provides a memory-efficient closed-form solution for applying this effect through an equivalent projection vector $\alpha$. The authors develop an efficient eigenvalue-based null-space approximation using a Prefix-Suffix Sum Ratio to handle large activations in LLMs, and demonstrate consistent improvements across multiple models and PTQ baselines, including 2- and 3-bit quantization and weight-activation PTQ. Key findings show substantial speedups over SVD-based methods, robust gains across generation and reasoning tasks, and superior performance of the closed-form solution over backpropagation-based optimization. Collectively, this work introduces a novel lens for PTQ in LLMs and provides practical, scalable techniques to push toward more aggressively quantized yet accurate models.
Abstract
Existing post-training quantization methods for large language models (LLMs) offer remarkable success. However, the increasingly marginal performance gains suggest that existing quantization strategies are insufficient to support the development of more compressed models. To inspire new directions for future research, this paper introduces the concept of null space into LLMs quantization. We argue that the quantization error can be effectively alleviated by constraining the post-quantization weight perturbation to lie within the null space of input activations. To prove this idea, we propose a plug-and-play null space projection module for existing milestone PTQ baselines named Q2N. Specifically, we first design an efficient and accurate null space projection approximation method tailored to the characteristics of LLMs. Subsequently, we theoretically derive a closed-form solution for an equivalent vector of the obtained projection matrix, which satisfies practical inference condition while avoiding additional memory overhead. Extensive experiments are conducted on various state-of-the-art LLMs (LLaMA3, DeepSeek, Qwen3) and baselines, demonstrating the effectiveness of both our Q2N and the perspective of null space optimization for LLMs quantization. We view this paper the first step to further alleviate the quantization error based on the insights of null space, hoping it inspiring future researchers to design more advanced quantization methods. Codes are available at https://github.com/zjq0455/q2n.