Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs

Paul Jonas Kurz, Tobias Jan Wieczorek, Mohamed A. Abdelsalam, Rahaf Aljundi, Marcus Rohrbach

TL;DR

Overall, this work presents the first systematic study linking quantization and reliability in multimodal settings, and finds that PTQ degrades both accuracy and reliability.

Abstract

Multimodal Large Language Models (MLLM) are increasingly deployed in domains where both reliability and efficiency are critical. However, current models remain overconfident, producing highly certain but incorrect answers. At the same time, their large size limits deployment on edge devices, necessitating compression. We study the intersection of these two challenges by analyzing how Post-Training Quantization (PTQ) compression affects both accuracy and reliability in Visual Question Answering (VQA). We evaluate two MLLMs, Qwen2-VL-7B and Idefics3-8B, quantized with data-free (HQQ) and data-aware (MBQ) methods across multiple bit widths. To counteract the reduction in reliability caused by quantization, we adapt the Selector confidence estimator for quantized multimodal settings and test its robustness across various quantization levels and out-of-distribution (OOD) scenarios. We find that PTQ degrades both accuracy and reliability. Data-aware methods soften the effect thereof. The Selector substantially mitigates the reliability impact. The combination of int4 MBQ and the Selector achieves the best efficiency-reliability trade-off, closing in on uncompressed performance at approx. 75% less memory demand. Overall, we present the first systematic study linking quantization and reliability in multimodal settings.

Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs

TL;DR

Overall, this work presents the first systematic study linking quantization and reliability in multimodal settings, and finds that PTQ degrades both accuracy and reliability.

Abstract

Multimodal Large Language Models (MLLM) are increasingly deployed in domains where both reliability and efficiency are critical. However, current models remain overconfident, producing highly certain but incorrect answers. At the same time, their large size limits deployment on edge devices, necessitating compression. We study the intersection of these two challenges by analyzing how Post-Training Quantization (PTQ) compression affects both accuracy and reliability in Visual Question Answering (VQA). We evaluate two MLLMs, Qwen2-VL-7B and Idefics3-8B, quantized with data-free (HQQ) and data-aware (MBQ) methods across multiple bit widths. To counteract the reduction in reliability caused by quantization, we adapt the Selector confidence estimator for quantized multimodal settings and test its robustness across various quantization levels and out-of-distribution (OOD) scenarios. We find that PTQ degrades both accuracy and reliability. Data-aware methods soften the effect thereof. The Selector substantially mitigates the reliability impact. The combination of int4 MBQ and the Selector achieves the best efficiency-reliability trade-off, closing in on uncompressed performance at approx. 75% less memory demand. Overall, we present the first systematic study linking quantization and reliability in multimodal settings.
Paper Structure (13 sections, 1 equation, 1 figure, 2 tables)

This paper contains 13 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Methodology
  4. Selective VQA with Multimodal LLMs
  5. MaxProb (Intrinsic Baseline).
  6. Selector (Extrinsic Estimator).
  7. Quantization Methods
  8. Experimental Setup
  9. Results and Analysis
  10. RQ1: Effect of Quantization
  11. RQ2: Selector Compensation
  12. RQ3: Out-of-Distribution Selector Robustness
  13. Discussion and Conclusion

Figures (1)

  • Figure 1: Evolution of VQA accuracy and coverage across quantizations. 0;0 denotes full in-distribution data from VQAv2. OOD data is sourced from AdVQA and VizWiz. The opaque transition from 100;0 to 0;100 reflects the fundamental multimodal shift caused by a transition towards conversational questions and lower-quality images. Top is Idefics3-8B, bottom is Qwen2-VL-7B.