SPEED-Q: Staged Processing with Enhanced Distillation towards Efficient Low-bit On-device VLM Quantization

TL;DR

SPEED-Q addresses the challenge of deploying multi-modal VLMs with 1–2B parameters on edge devices by introducing a staged quantization strategy that decouples ViT and LLM quantization, paired with distillation-enhanced training to stabilize low-bit optimization. The approach combines group-wise and bilevel quantization, asymmetric clipping initialization, self-distillation, and multi-loss optimization to achieve accurate 2/4-bit weight-only quantization for both vision and language modules. Empirical results across diverse multimodal benchmarks show SPEED-Q consistently surpasses state-of-the-art on-device VLMs, reducing model size and memory footprint while maintaining competitive or superior accuracy, including strong performance at 2-bit settings. These advances significantly improve the practicality of deploying high-performing VLMs on resource-constrained devices, with demonstrated scalability to larger models and concrete efficiency gains for edge inference.

Abstract

Deploying Vision-Language Models (VLMs) on edge devices (e.g., smartphones and robots) is crucial for enabling low-latency and privacy-preserving intelligent applications. Given the resource constraints of these devices, quantization offers a promising solution by improving memory efficiency and reducing bandwidth requirements, thereby facilitating the deployment of VLMs. However, existing research has rarely explored aggressive quantization on VLMs, particularly for the models ranging from 1B to 2B parameters, which are more suitable for resource-constrained edge devices. In this paper, we propose SPEED-Q, a novel Staged Processing with Enhanced Distillation framework for VLM low-bit weight-only quantization that systematically addresses the following two critical obstacles: (1) significant discrepancies in quantization sensitivity between vision (ViT) and language (LLM) components in VLMs; (2) training instability arising from the reduced numerical precision inherent in low-bit quantization. In SPEED-Q, a staged sensitivity adaptive mechanism is introduced to effectively harmonize performance across different modalities. We further propose a distillation-enhanced quantization strategy to stabilize the training process and reduce data dependence. Together, SPEED-Q enables accurate, stable, and data-efficient quantization of complex VLMs. SPEED-Q is the first framework tailored for quantizing entire small-scale billion-parameter VLMs to low bits. Extensive experiments across multiple benchmarks demonstrate that SPEED-Q achieves up to 6x higher accuracy than existing quantization methods under 2-bit settings and consistently outperforms prior on-device VLMs under both 2-bit and 4-bit settings. Our code and models are available at https://github.com/antgroup/SPEED-Q.

Figures (9)

• Figure 1: Comparison with SOTA on-device VLMs. The proposed quantized version of InternVL3 outperforms the existing on-device VLMs with a smaller model file size.
• Figure 2: The average absolute gradients of vision (ViT) and language (LLM) module in the InternVL2.5-1B quantization-aware training process.
• Figure 3: Pipeline of the staged quantization strategy. (a) Stage 1: The less sensitive ViT is quantized using an image-only calibration set. (b) Stage 2: Only the projector is trained to better align the quantized ViT (qViT) and the LLM. (c) Stage 3: qViT is frozen, and both the projector and the more sensitive LLM undergo quantization-aware training.
• Figure 4: Illustration of the distillation-enhanced quantization. Asymmetric clipping provides a more favorable initialization for quantization-aware training. Our objective combines forward KL, reverse KL, and a task-specific loss with ground-truth labels. The self-distillation framework, together with this multi-loss optimization, improves training stability and reduces dependence on large calibration datasets.
• Figure 5: On-device efficiency evaluation of FP16 and low-bit InternVL2.5-1B on Samsung Galaxy S25 Ultra.