LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation

TL;DR

LoPRo addresses the challenge of high-accuracy, fine-tuning-free weight-only PTQ for large language models in the sub-3-bit regime. It combines a permutation-based partial rotation of the residual after low-rank decomposition with a light Rank-1 SVD sketching method (R1SVD) to reduce quantization cost and memory. The method achieves state-of-the-art accuracy at 2-bit and 3-bit quantization on LLaMA-2/3 and Mixtral-8x7B, while delivering substantial speedups and low overhead compared to finetuning-based baselines. By decoupling the quantized residual from the low-rank component and leveraging structured rotation, LoPRo offers a practically efficient route to deploying large models on resource-constrained hardware, with clear avenues for extending to activation quantization and further acceleration.

Abstract

Post-training quantization (PTQ) enables effective model compression while preserving relatively high accuracy. Current weight-only PTQ methods primarily focus on the challenging sub-3-bit regime, where approaches often suffer significant accuracy degradation, typically requiring fine-tuning to achieve competitive performance. In this work, we revisit the fundamental characteristics of weight quantization and analyze the challenges in quantizing the residual matrix under low-rank approximation. We propose LoPRo, a novel fine-tuning-free PTQ algorithm that enhances residual matrix quantization by applying block-wise permutation and Walsh-Hadamard transformations to rotate columns of similar importance, while explicitly preserving the quantization accuracy of the most salient column blocks. Furthermore, we introduce a mixed-precision fast low-rank decomposition based on rank-1 sketch (R1SVD) to further minimize quantization costs. Experiments demonstrate that LoPRo outperforms existing fine-tuning-free PTQ methods at both 2-bit and 3-bit quantization, achieving accuracy comparable to fine-tuning baselines. Specifically, LoPRo achieves state-of-the-art quantization accuracy on LLaMA-2 and LLaMA-3 series models while delivering up to a 4$\times$ speedup. In the MoE model Mixtral-8x7B, LoPRo completes quantization within 2.5 hours, simultaneously reducing perplexity by 0.4$\downarrow$ and improving accuracy by 8\%$\uparrow$. Moreover, compared to other low-rank quantization methods, LoPRo achieves superior accuracy with a significantly lower rank, while maintaining high inference efficiency and minimal additional latency.

Figures (9)

• Figure 1: In LoPRo, the permutation-partial rotation effectively relieves the quantization burden of the residual matrix. Moreover, the R1SVD low-rank approximation maintains high efficiency and compression ratio. Specifics: (A). Using a real matrix and its variants to illustrate the data characteristics and transformations in LoPRo process;(B). A layer of corresponding empirical distribution in LoPRo and more visualization shown in Appendix \ref{['appendix_visual']}; (C). The Inference workflow of LoPRo.
• Figure 2: Visualization of layers in LLaMA2-7B
• Figure 3: Visualization of layers in LLaMA2-7B
• Figure 4: Visualization of layers in LLaMA2-13B
• Figure 5: Visualization of layers in LLaMA2-13B