Towards Understanding Best Practices for Quantization of Vision-Language Models

TL;DR

This work tackles the resource demands of multimodal LLMs by examining weight quantization strategies applied to BLIP-2 and LLaVA. It systematically analyzes component sensitivity across ViT, Q-Former, and LLM in retrieval, captioning, and VQA tasks, using uniform PTQ and state-of-the-art methods GPTQ and AWQ. Key findings include that LLMs are often the most sensitive to quantization, AWQ tends to prioritize LLM preservation while GPTQ distributes impact more evenly, and high accuracy can be maintained at low bit-widths on the order of $3.5$–$4.5$ bpw. The study provides practical deployment guidance and a reproducible framework for component-wise quantization in multimodal systems, with open-source calibration tools to support further research.

Abstract

Large language models (LLMs) deliver impressive results for a variety of tasks, but state-of-the-art systems require fast GPUs with large amounts of memory. To reduce both the memory and latency of these systems, practitioners quantize their learned parameters, typically at half precision. A growing body of research focuses on preserving the model performance with more aggressive bit widths, and some work has been done to apply these strategies to other models, like vision transformers. In our study we investigate how a variety of quantization methods, including state-of-the-art GPTQ and AWQ, can be applied effectively to multimodal pipelines comprised of vision models, language models, and their connectors. We address how performance on captioning, retrieval, and question answering can be affected by bit width, quantization method, and which portion of the pipeline the quantization is used for. Results reveal that ViT and LLM exhibit comparable importance in model performance, despite significant differences in parameter size, and that lower-bit quantization of the LLM achieves high accuracy at reduced bits per weight (bpw). These findings provide practical insights for efficient deployment of MLLMs and highlight the value of exploration for understanding component sensitivities in multimodal models. Our code is available at https://github.com/gautomdas/mmq.

Figures (12)

• Figure 1: Highlighted analysis. In this work, we investigate how quantizations affects multimodal models on a variety of tasks, specifically BLIP-2 (left) and LLaVA (right). We compare state-of-the-art quantization strategies (left) and also discuss which portions of the pipeline are most amenable to quantization (right).
• Figure 2: Uniform Quantization Experiments
• Figure 3: Uniform quantization impact on COCO captioning for BLIP-2. We focus these plots on the high-quality range (see appendix for full results), drawing attention first to which model component we quantize (left), then layer type (middle), then group (right). We provide 2 black stars on each plot, indicating the results for quantizing the entire pipeline at 8 and 16 bits. The best performance-size tradeoffs tend to come when quantizing the entire pipeline.
• Figure 4: SOTA Unimodal Quantization Performance-Size Tradeoff for retrieval, captioning, and VQA tasks. We apply GPTQ and AWQ to entire model components (vision encoder, connector, LLM), of BLIP-2 and LLaVA. Unimodal SOTA methods are able to preserve full-precision performance at high compression rates, even in the multimodal context.
• Figure 5: Individual quantization of BLIP-2's ViT, Q-Former, and LLM for VQAv2 and GQA. We show that the different components have varying sensitivities in regards to bit width and quantization method. The LLM tends to have the highest sensitivity in question-answering tasks, followed by the ViT and Q-Former.