Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation

TL;DR

_loRDS_ addresses the expressivity gap in block-wise LLM quantization by introducing a continuous low-rank scaling $S \approx BA$, enabling a unified PTQ, QAT, and quantized PEFT lifecycle with zero inference overhead. The method initializes from standard block-wise statistics and refines via alternating optimization to minimize $\|W - (BA) \odot Q\|_F^2$, while enabling differentiable QAT through an STE-based path and a multiplicative PEFT mechanism with $\Delta W = Q \odot (B'A' - BA)$. Empirically, LoRDS achieves state-of-the-art or competitive performance across PTQ, QAT, and PEFT on model families like Llama3-8B and Qwen3-8B/4B, delivering significant improvements in reconstruction accuracy, downstream tasks, and with substantial throughput gains on GPUs using optimized Triton kernels (e.g., up to 1.5x speedup over QLoRA on RTX 4090). Overall, this work provides an integrated, high-expressivity, low-overhead solution for unified compression and adaptation of large language models, breaking traditional block boundaries and enabling high-rank updates within a low-rank budget.

Abstract

Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demonstrate that element-wise quantization can be made as efficient as block-wise scaling while providing strictly superior expressive power by modeling the scaling manifold as continuous low-rank matrices ($S = BA$). We propose Low-Rank Decomposed Scaling (LoRDS), a unified framework that rethinks quantization granularity through this low-rank decomposition. By "breaking the blocks" of spatial constraints, LoRDS establishes a seamless efficiency lifecycle: it provides high-fidelity PTQ initialization refined via iterative optimization, enables joint QAT of weights and scaling factors, and facilitates high-rank multiplicative PEFT adaptation. Unlike additive PEFT approaches such as QLoRA, LoRDS enables high-rank weight updates within a low-rank budget while incurring no additional inference overhead. Supported by highly optimized Triton kernels, LoRDS consistently outperforms state-of-the-art baselines across various model families in both quantization and downstream fine-tuning tasks. Notably, on Llama3-8B, our method achieves up to a 27.0% accuracy improvement at 3 bits over NormalFloat quantization and delivers a 1.5x inference speedup on NVIDIA RTX 4090 while enhancing PEFT performance by 9.6% on downstream tasks over 4bit QLoRA, offering a robust and integrated solution for unified compression and adaptation of LLMs.

Figures (3)

• Figure 1: Architectural comparison between (a) QLoRA and (b) our proposed LoRDS. (a) QLoRA employs standard block-wise quantization where parameters within each block share a static scaling factor. During fine-tuning, it introduces an auxiliary, full-precision additive adapter ($W + BA$), which cannot be merged into the quantized weights, leading to additional inference latency. (b) LoRDS fundamentally rethinks the scaling mechanism by decomposing the block-wise scaling matrix $S$ into a low-rank product $BA$. This multiplicative formulation ($W \odot BA$) serves a dual purpose: it can be refined via PTQ or QAT to achieve more granular scaling, or directly fine-tuned for PEFT. LoRDS enables high-rank parameter updates and incurs zero additional inference overhead, as the learned scaling factors are naturally absorbed into the dequantization process.
• Figure 2: Comparison of operator latency between bitsandbytes NF4, peft QLoRA, and LoRDS (Ours) across various hardware platforms (RTX 4090, RTX 5090, and H800). The x-axis represents the total number of processed tokens $M$.
• Figure 3: Singular value distribution of the weight update $\Delta W$ for the first $Q$ projection layer of Llama3-8B. While the rank of QLoRA is strictly constrained by its additive design, LoRDS achieves full-rank updates similar to full fine-tuning through its multiplicative scaling manifold.