Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients

Ziwei Xiang, Fanhu Zeng, Hongjian Fang, Rui-Qi Wang, Renxing Chen, Yanan Zhu, Yi Chen, Peipei Yang, Xu-Yao Zhang

Abstract

Large Vision Language Models (LVLMs) have achieved remarkable success in a range of downstream tasks that require multimodal interaction, but their capabilities come with substantial computational and memory overhead, which hinders practical deployment. Among numerous acceleration techniques, post-training quantization is a popular and effective strategy for reducing memory cost and accelerating inference. However, existing LVLM quantization methods typically measure token sensitivity at the modality level, which fails to capture the complex cross-token interactions and falls short in quantitatively measuring the quantization error at the token level. As tokens interact within the model, the distinction between modalities gradually diminishes, suggesting the need for fine-grained calibration. Inspired by axiomatic attribution in mechanistic interpretability, we introduce a fine-grained quantization strategy on Quantization-aware Integrated Gradients (QIG), which leverages integrated gradients to quantitatively evaluate token sensitivity and push the granularity from modality level to token level, reflecting both inter-modality and intra-modality dynamics. Extensive experiments on multiple LVLMs under both W4A8 and W3A16 settings show that our method improves accuracy across models and benchmarks with negligible latency overhead. For example, under 3-bit weight-only quantization, our method improves the average accuracy of LLaVA-onevision-7B by 1.60%, reducing the gap to its full-precision counterpart to only 1.33%. The code is available at https://github.com/ucas-xiang/QIG.

Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients

Abstract

Large Vision Language Models (LVLMs) have achieved remarkable success in a range of downstream tasks that require multimodal interaction, but their capabilities come with substantial computational and memory overhead, which hinders practical deployment. Among numerous acceleration techniques, post-training quantization is a popular and effective strategy for reducing memory cost and accelerating inference. However, existing LVLM quantization methods typically measure token sensitivity at the modality level, which fails to capture the complex cross-token interactions and falls short in quantitatively measuring the quantization error at the token level. As tokens interact within the model, the distinction between modalities gradually diminishes, suggesting the need for fine-grained calibration. Inspired by axiomatic attribution in mechanistic interpretability, we introduce a fine-grained quantization strategy on Quantization-aware Integrated Gradients (QIG), which leverages integrated gradients to quantitatively evaluate token sensitivity and push the granularity from modality level to token level, reflecting both inter-modality and intra-modality dynamics. Extensive experiments on multiple LVLMs under both W4A8 and W3A16 settings show that our method improves accuracy across models and benchmarks with negligible latency overhead. For example, under 3-bit weight-only quantization, our method improves the average accuracy of LLaVA-onevision-7B by 1.60%, reducing the gap to its full-precision counterpart to only 1.33%. The code is available at https://github.com/ucas-xiang/QIG.
Paper Structure (21 sections, 19 equations, 7 figures, 9 tables)

This paper contains 21 sections, 19 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision Language Models
  4. Post-Training Quantization
  5. Interpretability and Token Sensitivity
  6. Method
  7. Preliminaries
  8. Sensitivity Differences Between Modalities and Tokens
  9. Fine-Grained Quantization
  10. Experiment
  11. Experimental Setup
  12. Main Results
  13. Ablation Study and Further Analysis
  14. Conclusion
  15. Proof of Quantization-Aware Integrated Gradients Completeness
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Token-level quantization sensitivity across layers in the form of heatmap and curves. At layers 1 and 16, we show both the token-level sensitivity heatmap and its channel-averaged line curve for special, vision, and text tokens, measured using our Quantization-aware Integrated Gradients (QIG).
  • Figure 2: Visualization of activation distributions in InternVL2-8B during calibration. We visualize two representative layers and four linear sub-layers. In each panel, the horizontal axis denotes token positions in the multimodal sequence and the vertical axis indexes hidden channels; color encodes the average activation magnitude per token–channel pair over the calibration set. The plots reveal four recurring phenomena: massive activations, layer heterogeneity, sub-layer divergence, and token variability. These patterns indicate that coarse modality-level sensitivity modeling is insufficient, motivating our token-level sensitivity weighting.
  • Figure 3: Comparison between modality-balanced quantization and our fine-grained quantization. Different colors indicate token types. Unlike MBQ, which assigns modality-level sensitivity, our method computes token-level sensitivity via Quantization-aware Integrated Gradients (QIG) during calibration, enabling more effective quantization.
  • Figure A1: The baseline fails to identify the film and produces an incomplete answer, whereas our fine-grained quantization successfully preserves the correct semantic prediction and matches the full-precision model.
  • Figure A2: The baseline fails to answer the question and provides no reasoning, whereas our fine-grained quantization preserves both correctness and detailed visual justification, closely matching the full-precision model.
  • ...and 2 more figures