Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization

TL;DR

Quantizing large language models is hindered by a small set of highly sensitive outlier weights. NPFT identifies these outliers via a Fisher-information proxy, estimates the Hessian trace through random perturbations, and performs LoRA-based PEFT fine-tuning with noise to minimize the expected loss under perturbation, thereby reducing $\mathrm{Tr}(H)$ and enabling uniform-precision quantization. The method achieves consistent improvements across uniform and non-uniform quantizers, matches or surpasses strong PTQ baselines on OPT and LLaMA, and delivers notable inference efficiency gains with lower training cost than QAT approaches. This approach enhances hardware-friendly deployment by removing the need to preserve outliers in higher precision while maintaining or improving quantized-model performance.

Abstract

Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose Noise Perturbation Fine-tuning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.

Figures (5)

• Figure 1: (Left) RTN suffers from the degradation caused by quantizing outliers. Preserving 0.5% of the outliers in FP16 can greatly recover the performance of the quantized model. (Right) NPFT brings significant improvement to the performance of single bit-width models without preserving any outliers in FP16. When applied to OPT 1.3B/2.7B models, our method outperforms RTN baseline by a large PPL margin of over 2.9/0.9 on the C4 benchmark.
• Figure 2: (Left) Existing PTQ methods preserve outliers in FP16 to prevent significant performance degradation, but the mixed-precision format is not hardware-friendly. (Right) Our proposed NPFT regularizes the outliers' sensitivity through efficient fine-tuning, which enhances the performance of single bit-width quantized models without special treatment to the outliers.
• Figure 3: Visualization of outlier sensitivity in OPT-1.3B. The sensitivity is obtained by calculating $\mathcal{F}$. After fine-tuning, the outliers become less sensitive compared to the original model.
• Figure 4: Impact of perturbation ratio on model performance.
• Figure 5: Training loss curves of OPT-1.3B. The perturbed model converges more swiftly with $\beta >0$.