Zero-Shot Quantization via Weight-Space Arithmetic

Abstract

We show that robustness to post-training quantization (PTQ) is a transferable direction in weight space. We call this direction the quantization vector: extracted from a donor task by simple weight-space arithmetic, it can be used to patch a receiver model and improve robustness to PTQ-induced noise by as much as 60%, without receiver-side quantization-aware training (QAT). Because the method requires no receiver training data, it provides a zero-shot, low-cost alternative to QAT for extremely low-bit deployment. We demonstrate this on Vision Transformer (ViT) models. More broadly, our results suggest that quantization robustness is not merely a byproduct of task-specific training, but a reusable feature of weight-space geometry that can be transferred rather than retrained.

Figures (5)

• Figure 1: Zero-shot quantization vector patching. A donor quantization vector $\rho_{\mathcal{D}} \coloneq \theta_{\mathcal{D},\text{QAT}} - \theta_{\mathcal{D}}$, extracted as the weight-space displacement between a standard fine-tuned donor checkpoint and its QAT counterpart, is added to a receiver checkpoint to obtain the patched model $\theta_{\mathcal{R} \leftarrow \mathcal{D}} = \theta_{\mathcal{R}} + \lambda \rho_{\mathcal{D}}$. The schematic situates vanilla PTQ, QV patching, and full QAT in the computational cost–3-bit accuracy plane, showing that QV patching can substantially improve low-bit robustness over PTQ while avoiding the full cost of receiver-side QAT.
• Figure 2: Quantization vector transferability for ViT/B-16. Top-1 accuracy change ($\Delta$) from patching receiver $r$ with donor $d$ quantization vector, relative to vanilla 3-bit PTQ. Left shows transfer with a constant scaling factor, while right demonstrates that modulating the magnitude $\lambda$ eliminates destructive interference and maximizes gains.
• Figure 3: Overview of the PTQ vs. QAT decision pipeline.
• Figure 4: Quantization vector transferability for ViT/T-16. Top-1 accuracy change ($\Delta$) from patching receiver $r$ with donor $d$ quantization vector, relative to vanilla 3-bit PTQ. Left shows transfer with a constant scaling factor, while right demonstrates that modulating the magnitude $\lambda$ eliminates destructive interference and maximizes gains.
• Figure 5: Quantization vector transferability for ViT/L-16. Top-1 accuracy change ($\Delta$) from patching receiver $r$ with donor $d$ quantization vector, relative to vanilla 3-bit PTQ. Left shows transfer with a constant scaling factor, while right demonstrates that modulating the magnitude $\lambda$ eliminates destructive interference and maximizes gains.