Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models

Huijie Guo, Jingyao Wang, Lingyu Si, Jiahuan Zhou, Changwen Zheng, Wenwen Qiang

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have shown impressive reasoning capabilities across vision-language tasks, yet still face the challenge of compute-difficulty mismatch. Through empirical analyses, we identify that existing decoding methods may waste compute on easy cases while underserving hard ones, affecting both model effectiveness and efficiency. To address this issue, we first develop a theoretical framework that links sampling coverage, instance difficulty, and residual risk. Our analysis reveals that multimodal reasoning exhibits a heavy-tailed difficulty distribution; a small subset of hard or ambiguous samples dominates the residual failure probability. Based on this insight, we propose Coverage-Aware Multimodal Decoding (CAMD), an adaptive inference mechanism that dynamically allocates computation according to estimated uncertainty. CAMD integrates evidence-weighted scoring, posterior coverage estimation, and sequential Bayesian updating to balance efficiency and reliability under a limited token budget. Experiments on various benchmark datasets and baselines demonstrate the effectiveness and advantages of our approach.

CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have shown impressive reasoning capabilities across vision-language tasks, yet still face the challenge of compute-difficulty mismatch. Through empirical analyses, we identify that existing decoding methods may waste compute on easy cases while underserving hard ones, affecting both model effectiveness and efficiency. To address this issue, we first develop a theoretical framework that links sampling coverage, instance difficulty, and residual risk. Our analysis reveals that multimodal reasoning exhibits a heavy-tailed difficulty distribution; a small subset of hard or ambiguous samples dominates the residual failure probability. Based on this insight, we propose Coverage-Aware Multimodal Decoding (CAMD), an adaptive inference mechanism that dynamically allocates computation according to estimated uncertainty. CAMD integrates evidence-weighted scoring, posterior coverage estimation, and sequential Bayesian updating to balance efficiency and reliability under a limited token budget. Experiments on various benchmark datasets and baselines demonstrate the effectiveness and advantages of our approach.
Paper Structure (17 sections, 1 theorem, 16 equations, 6 figures, 2 tables)

This paper contains 17 sections, 1 theorem, 16 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Decoding Strategies for MLLMs
  5. Problem Settings and Analysis
  6. Problem Settings
  7. Emprical Analysis
  8. Methodology
  9. Theoretical Foundations
  10. Coverage-Aware Multimodal Decoding
  11. Evidence-weighted scoring
  12. Posterior coverage estimation
  13. Bayesian adaptive sampling
  14. Experiments
  15. Experimental Settings
  16. ...and 2 more sections

Key Result

Theorem 4.2

Let $g(s)$ denote the density of $G(s)$ near $s\to0$. Then the asymptotic decay of $\Delta(K)$ follows:

Figures (6)

  • Figure 1: Reasoning complexity across multimodal tasks.
  • Figure 2: Motivating experiments on MathVista. (a) Accuracy (%) vs. average generated tokens; (b) Cross-modal consistency (%) vs. tokens; (c) P95 latency (ms) vs. tokens; (d) Accuracy (%) vs. average samples.
  • Figure 3: The framework of the proposed CAMD (Coverage-Aware Multimodal Decoding) for MLLMs.
  • Figure 4: Sequence Length vs. Performance with different decoding methods on POPE-R (left) and MSRVTT-QA (right).
  • Figure 5: Performance of different methods on GPT-4 assisted evaluation. Note that some values were not reported in the original papers; we have replaced them with "-" in the figure.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 4.1: $\delta$-coverage sampling size $N_\delta$
  • Theorem 4.2: Tail-dominated convergence rate