VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models

TL;DR

VEQ tackles the problem of compressing MoE Vision-Language Models with post-training quantization by explicitly addressing cross-modal heterogeneity and expert routing sparsity. It introduces two components, Modality-Expert-Aware Quantization (VEQ-ME) and Modality-Affinity-Aware Quantization (VEQ-MA), with a weighted expert loss $\mathcal{L}_{\text{Weighted}}$ and a reweighted Hessian $\tilde{H}$ to preserve important tokens. Empirical results on Kimi-VL-Instruct and Qwen3-VL-30B-Instruct show substantial gains under aggressive 3-bit quantization (W3A16), e.g., average improvements of $2.04\%$ and $3.09\%$ respectively, and a TextVQA improvement of $21.4\%$ on one setup. Ablation studies demonstrate the necessity of both components and their hyperparameters, confirming that modality-aware prioritization and affinity weighting are key for robust low-bit quantization of MoE VLMs, enabling practical deployment on resource-constrained platforms.

Abstract

Mixture-of-Experts(MoE) Vision-Language Models (VLMs) offer remarkable performance but incur prohibitive memory and computational costs, making compression essential. Post-Training Quantization (PTQ) is an effective training-free technique to address the massive memory and computation overhead. Existing quantization paradigms fall short as they are oblivious to two critical forms of heterogeneity: the inherent discrepancy between vision and language tokens, and the non-uniform contribution of different experts. To bridge this gap, we propose Visual Expert Quantization (VEQ), a dual-aware quantization framework designed to simultaneously accommodate cross-modal differences and heterogeneity between experts. Specifically, VEQ incorporates 1)Modality-expert-aware Quantization, which utilizes expert activation frequency to prioritize error minimization for pivotal experts, and 2)Modality-affinity-aware Quantization, which constructs an enhanced Hessian matrix by integrating token-expert affinity with modality information to guide the calibration process. Extensive experiments across diverse benchmarks verify that VEQ consistently outperforms state-of-the-art baselines. Specifically, under the W3A16 configuration, our method achieves significant average accuracy gains of 2.04\% on Kimi-VL and 3.09\% on Qwen3-VL compared to the previous SOTA quantization methods, demonstrating superior robustness across various multimodal tasks. Our code will be available at https://github.com/guangshuoqin/VEQ.

Figures (10)

• Figure 1: Zero-shot performance of Kimi-VL-Instruct under 3-bit weight quantization (W3A16). Our methods consistently outperform established baselines, demonstrating superior robustness.
• Figure 2: Comparative analysis of activation characteristics across different modalities. Peaks represent high activation frequency.
• Figure 3: Overview of the proposed VEQ framework. Our method consists of two core components: (1) VEQ-ME, which dynamically assigns importance scores $S_i$ to experts based on their activation frequencies, thereby prioritizing error minimization for pivotal experts in the reconstruction loss; and (2) VEQ-MA, which constructs an enhanced Hessian matrix by integrating token-expert affinity scores and modality sensitivity, enabling the calibration process to adapt to the varying sensitivities of multi-modal tokens.
• Figure 4: Analysis of gradient magnitude across 128 samples from the COCO lin2014coco dataset. The text tokens exhibit significantly higher gradient norms compared to vision tokens, with an average ratio of 22.4.
• Figure 5: Detailed gradient analysis of a representative sample (Sample 88). The visualization highlights that the text-to-vision gradient ratio reaches approximately 15, confirming the dominance of textual information in the inference process.