FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching

TL;DR

FLRQ addresses the inefficiency of fixed-rank post-training quantization for large language models by introducing a flexible, data-driven rank selection method (R1-FLR) and an iterative clipping-based refinement (BLC). By replacing standard SVD with the rank-1 sketch, FLRQ dramatically reduces computation while enabling per-layer rank adaptation, and its clipping-based optimization minimizes quantization error under calibration. Empirically, FLRQ achieves state-of-the-art or near-state-of-the-art quantization quality across 2–4 bit regimes on OPT and LLaMA family models, with substantially lower memory overhead and comparable or smaller latency due to kernel fusion. These contributions enable robust, efficient low-bit quantization of very large models, improving deployability on resource-constrained hardware without retraining or substantial fine-tuning.

Abstract

Traditional post-training quantization (PTQ) is considered an effective approach to reduce model size and accelerate inference of large-scale language models (LLMs). However, existing low-rank PTQ methods require costly fine-tuning to determine a compromise rank for diverse data and layers in large models, failing to exploit their full potential. Additionally, the current SVD-based low-rank approximation compounds the computational overhead. In this work, we thoroughly analyze the varying effectiveness of low-rank approximation across different layers in representative models. Accordingly, we introduce \underline{F}lexible \underline{L}ow-\underline{R}ank \underline{Q}uantization (FLRQ), a novel solution designed to quickly identify the accuracy-optimal ranks and aggregate them to achieve minimal storage combinations. FLRQ comprises two powerful components, Rank1-Sketch-based Flexible Rank Selection (R1-FLR) and Best Low-rank Approximation under Clipping (BLC). R1-FLR applies the R1-Sketch with Gaussian projection for the fast low-rank approximation, enabling outlier-aware rank extraction for each layer. Meanwhile, BLC aims at minimizing the low-rank quantization error under the scaling and clipping strategy through an iterative method. FLRQ demonstrates strong effectiveness and robustness in comprehensive experiments, achieving state-of-the-art performance in both quantization quality and algorithm efficiency.

Figures (13)

• Figure 1: The presentation at the model layer and internals of FLRQ, (1) to (3) represent the three steps of the FLRQ algorithm respectively. In detail, FLRQ utilizes flexible rank selections (1), activation-based scaling, and iterative Best Low-rank Approximation under Clipping (BLC) algorithm (2), leading to higher quantization accuracy and smaller model size (3).
• Figure 2: As the extraction rank increases, the curves of the relative error $\mathbb{E}$ under $L_2$ norm and the $amax$ value decrease.
• Figure 3: Comparison of Throughput and Latency between Baseline (W4A16) and FLRQ (W4A16+Lora).
• Figure 4: More visualization in LLaMA2-7b on the relationship between rank selection and the error $\mathbb{E}$/$absmax$, where the blue lines (- - -) stand for the optimal rank from R1-FLR. Results demonstrate how varying the rank affects the quantization error, providing insights into the trade-offs between different rank choices and their corresponding impact on quantization accuracy.
• Figure 5: The scaling laws of perplexity under bit-level.