WUSH: Near-Optimal Adaptive Transforms for LLM Quantization

TL;DR

This work tackles the challenge of quantizing LLM weights and activations in the presence of heavy-tailed outliers by introducing WUSH, a data-aware blockwise transform that complements a Hadamard backbone with a second-order statistics-driven component. The authors derive closed-form optimal transforms for both floating-point and, asymptotically, integer quantization, under a probabilistic, data-calibration framework and a block-diagonal constraint. The method achieves lower quantization loss than fixed Hadamard approaches across multiple formats and models, with experimental results on layer-wise RTN losses and LLМ benchmarks demonstrating improved accuracy and recovery. The approach offers a principled, calibration-based alternative to existing heuristics and points to practical directions for integration with GPTQ and hardware-efficient kernel design. Overall, WUSH provides a theoretically grounded, implementable path to near-optimal adaptive transforms for LLM quantization that narrows the gap to full-precision performance.

Abstract

Quantization to low bitwidth is a standard approach for deploying large language models, however, a few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer. A common mitigation approach is to apply some fixed orthogonal transforms, such as Hadamard matrices, before quantization, which typically reduces the dynamic range. Yet, these transforms ignore the statistics of the data, and their optimality is currently not understood. In this work, we derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization using standard data-free quantizers for common numerical formats. Specifically, we provide derivations of the optimal adaptive (data-aware) transforms for round-to-nearest (RTN), AbsMax-scaled block quantizers for both integer and floating-point formats. The resulting construction, which we call WUSH, combines a Hadamard backbone with a data-dependent component based on second-order moments, yielding a non-orthogonal transform that is provably optimal under mild assumptions and remains structured for efficient implementation. Preliminary experimental results show that our approach consistently improves upon the Hadamard transform for common formats.

Figures (5)

• Figure 1: Comparison of different transforms on Qwen3-8B for both NVFP4 and MXFP4 RTN quantization on Platinum benchmark tasks. The left table shows accuracy results across the individual Platinum benchmark tasks, while the right plot shows the average accuracy scores together with their standard deviations for each transform.
• Figure 2: Comparison of different transforms on Qwen3-14B for both NVFP4 and MXFP4 RTN quantization on Platinum benchmark tasks. The left table shows accuracy results across the individual Platinum benchmark tasks, while the right plot shows the average accuracy scores together with their standard deviations for each transform.
• Figure 3: Comparison of different transforms on Llama3.2-3B Instruct for both NVFP4 and MXFP4 RTN quantization on Platinum benchmark tasks. The left table shows accuracy results across the individual Platinum benchmark tasks, while the right plot shows the average accuracy scores together with their standard deviations for each transform.
• Figure 4: Comparison of different transforms on Llama3.1-8B Instruct for both NVFP4 and MXFP4 RTN quantization on Platinum benchmark tasks. The left table shows accuracy results across the individual Platinum benchmark tasks, while the right plot shows the average accuracy scores together with their standard deviations for each transform.
• Figure : Compute WUSH Transforms and Pre-Quantize Weights

Theorems & Definitions (5)

• Theorem 1: Optimal Transform
• Theorem 2: FP Quantization Error Modeling
• Theorem 3: INT Quantization Error Modeling
• Lemma 4: Maximum Inequalities for Tail-Bounded Distributions
• Lemma 5: Expectation from the Survival Function