Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models

TL;DR

The paper addresses calibration bias in post-training quantization for vision-language models arising from token-level heterogeneity between visual and textual tokens. It introduces TLQ, a gradient-guided, token-level importance framework that constructs a compact calibration set and couples it with a quantization-exposed, multi-GPU layer-wise calibration that mirrors the actual inference path. Experiments on LLaVA-onevision and Qwen2-VL across three model scales and two quant settings show TLQ consistently improves FP16 accuracy retention over strong baselines and enables calibration on modest GPUs. The work offers practical, scalable PTQ solutions for deploying large multimodal models while maintaining stability and performance, and code will be released publicly.

Abstract

Post-training quantization (PTQ) is a primary approach for deploying large language models without fine-tuning, and the quantized performance is often strongly affected by the calibration in PTQ. By contrast, in vision-language models (VLMs), substantial differences between visual and text tokens in their activation distributions and sensitivities to quantization error pose significant challenges for effective calibration during PTQ. In this work, we rethink what PTQ calibration should align with in VLMs and propose the Token-level Importance-aware Layer-wise Quantization framework (TLQ). Guided by gradient information, we design a token-level importance integration mechanism for quantization error, and use it to construct a token-level calibration set, enabling a more fine-grained calibration strategy. Furthermore, TLQ introduces a multi-GPU, quantization-exposed layer-wise calibration scheme. This scheme keeps the layer-wise calibration procedure consistent with the true quantized inference path and distributes the complex layer-wise calibration workload across multiple RTX3090 GPUs, thereby reducing reliance on the large memory of A100 GPUs. TLQ is evaluated across two models, three model scales, and two quantization settings, consistently achieving performance improvements across all settings, indicating its strong quantization stability. The code will be released publicly.

Figures (3)

• Figure 1: Left: TLQ is compared with three representative PTQ baselines across two vision–language models with three model scales under two quantization settings. TLQ achieves the highest percentage of FP16 accuracy retained after quantization across all settings, demonstrating strong stability. Right: Illustration of the hardware efficiency of the proposed framework, which reduces the requirement from a single 80GB A100 GPU to a practical setup using three 24GB RTX3090 GPUs.
• Figure 2: Overview of our proposed Token-level Importance-aware Layer-wise Quantization framework. (a) Gradient-guided token selection identifies output-sensitive tokens for calibration, reducing token-agnostic bias from redundant tokens. (b) Quantization-exposed layer-wise calibration propagates quantized activations across layers to align calibration with the inference path. (c) Memory-efficient calibration under deployment constraints is achieved by decoupling layer-wise inference, scale computation, and loss evaluation across multiple GPUs, enabling calibration on three 24GB RTX3090 GPUs instead of a single 80GB A100.
• Figure 3: Visualization of token gradients of 13th transformer block in the LLaVA-onevision-7b on COCO caption dataset before(left) and after(right) our gradient-guided token selection. The blue regions correspond to gradients of visual tokens, the yellow regions represent gradients of textual tokens, and the white areas indicate zero gradients.