ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

TL;DR

ARCQuant introduces a novel NVFP4 quantization framework that maintains a strictly unified 4-bit data path by augmenting the input activations with quantized residual channels. The method maps the residual compensation into the extended reduction dimension of a single GEMM, enabling the use of optimized GEMM kernels and delivering W4A8-like fidelity under W4A4 hardware constraints. The authors provide a theoretical error bound showing the worst-case error is comparable to MXFP8 and demonstrate state-of-the-art accuracy on Llama 3.1-8B and Qwen2.5 families, with substantial end-to-end speedups on RTX 5090 and PRO 6000 GPUs. They also present an efficient fused kernel design and robust calibration, with open-source code available, offering a scalable, hardware-friendly pathway for high-throughput LLM inference at low precision.

Abstract

The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing Post-Training Quantization (PTQ) strategies to these formats: rotation-based methods compromise fine-grained block isolation; smoothing techniques struggle with significant 4-bit quantization errors; and mixed-precision approaches often conflict with hardware constraints on unified-precision computation. To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. This design integrates the error compensation process directly into the matrix reduction dimension, enabling the use of standard, highly optimized GEMM kernels with minimal overhead. Theoretical analysis confirms that the worst-case error bound of our dual-stage NVFP4 quantization is comparable to that of standard 8-bit formats such as MXFP8. Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3x speedup over FP16. Our code is available at https://github.com/actypedef/ARCQuant .

Figures (9)

• Figure 1: ARCQuant closes the NVFP4 accuracy gap while preserving high throughput on Blackwell platforms.
• Figure 2: Activation magnitudes (blue) and quantization errors (red) on Llama 3.1-8B o_proj. ARCQuant isolates outlier to effectively suppress errors via residual compensation, whereas Hadamard spreads outlier magnitudes.
• Figure 3: o_proj in Llama 3.1-8B, ARCQuant consistently suppressed MSE on NVFP4 across all layers.
• Figure 4: ARCQuant fuses main and residual computations by mapping them into the extended dimension.
• Figure 5: ARCQuant on a Transformers block in LLM.