Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs

TL;DR

Preserve-Then-Quantize introduces Structured Residual Reconstruction (SRR), a principled rank-allocation framework that splits the quantization-rank budget between preserving the dominant activation-scales of weights and reconstructing the remaining quantization error. By selecting an optimal split $k^*$ using a theory-guided surrogate that combines the spectral energy of $S\mathbf{W}$ with a random error proxy, SRR achieves a more faithful $\textbf{W} \approx \textbf{Q}+\textbf{L}\textbf{R}$ representation under low-bit post-training quantization. The framework extends naturally to Quantized PEFT (QPEFT), where SRR initializes a two-component adapter and applies gradient scaling on preserved directions to stabilize fine-tuning, yielding consistent gains across PTQ and QPEFT benchmarks. Empirical results show SRR delivering substantial perplexity reductions and improved zero-shot GLUE performance in PTQ, along with 5.9 percentage-point average gains in GLUE under 2-bit QPEFT, demonstrating robust, quantization-aware improvements across model families and quantizers. Overall, SRR provides a robust, scalable approach to leverage low-rank corrections by intelligently balancing structure preservation with error reconstruction.

Abstract

Quantization Error Reconstruction (QER) reduces accuracy loss in Post-Training Quantization (PTQ) by approximating weights as $\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$, using a rank-$r$ correction to reconstruct quantization error. Prior methods devote the full rank budget to error reconstruction, which is suboptimal when $\mathbf{W}$ has intrinsic low-rank structure and quantization corrupts dominant directions. We propose Structured Residual Reconstruction (SRR), a rank-allocation framework that preserves the top-$k$ singular subspace of the activation-scaled weight before quantization, quantizes only the residual, and uses the remaining rank $r-k$ for error reconstruction. We derive a theory-guided criterion for selecting $k$ by balancing quantization-exposed energy and unrecoverable error under rank constraints. We further show that resulting $\mathbf{Q} + \mathbf{L}\mathbf{R}$ parameterization naturally supports Quantized Parameter-Efficient Fine-Tuning (QPEFT), and stabilizes fine-tuning via gradient scaling along preserved directions. Experiments demonstrate consistent perplexity reductions across diverse models and quantization settings in PTQ, along with a 5.9 percentage-point average gain on GLUE under 2-bit QPEFT.

Figures (9)

• Figure 1: Preserve-then-quantize mechanism in Structured Residual Reconstruction (SRR), compared with standard QER. Top: In QER, quantizing the entire weight matrix destroys the intrinsic low-rank structure of the original weight, leaving errors that cannot be recovered by low-rank reconstruction. Bottom: In contrast, SRR preserves the dominant low-rank structure prior to quantization, preventing structural damage and resulting in a substantially smaller reconstruction error $\lVert \mathbf{W} - \mathbf{Q} - \mathbf{L}\mathbf{R} \rVert_F$ under the same rank budget.
• Figure 2: Alignment between reconstruction error and the rank-selection objective. Reconstruction error (Top) and surrogate objective (Bottom) as functions of the preserved rank $k$ under a rank budget of $r=64$. Similar trends across $k$ support the use of the objective for selecting $k^\star$. Results are shown for the Query (Left) and Output (Right) projections in LLaMA-2 7B, layer 10. Additional results for other projections are provided in Appendix \ref{['app:alignment_app']}.
• Figure 3: (a) Singular-value spectrum with the selected split $k^\star$. (b) Illustration of the corresponding low-rank factors, where gradients along the $k^\star$ preserved directions are scaled, while the remaining $r-k^\star$ residual directions are left unscaled.
• Figure 4: Training loss curves for QPEFT baselines on two GLUE tasks: STSB (Left) and CoLA (Right), over five epochs. Each method uses its best-performing learning rate (see Appendix \ref{['app:qpeft_experiment_details']}). SRR shows faster training loss reduction than the other baselines.
• Figure 5: Projection-wise distribution of the selected rank $k^{\star}$ under a rank budget of $r=64$. Each box plot summarizes layer-wise variations for (a) LLaMA-2 7B and (b) LLaMA-3.1 8B.