Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding

Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, Jiawei Zhou

TL;DR

This work tackles LVLM hallucinations by introducing Dropout Decoding, a training-free, inference-time strategy that quantifies visual-token uncertainty and masks uncertain tokens. Tokens are projected into the text space to produce per-token distributions, enabling epistemic-uncertainty–driven dropout and ensemble decoding over multiple masked contexts. The approach decomposes uncertainty into aleatoric and epistemic components and focuses on the latter to identify informative visual tokens, with majority voting across K masked decodings yielding more reliable outputs. Empirical results on CHAIR, THRONE, and MMBench demonstrate reduced object hallucinations and improved overall quality across multiple LVLM backbones, with modest computational overhead and scalable efficiency via batching. The method provides a principled, generalizable inference-time technique for enhancing the trustworthiness of multimodal generation without additional training data or fine-tuning.

Abstract

Large vision-language models (LVLMs) excel at multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. We present DROPOUT DECODING, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting it onto the text space and decomposing it into aleatoric and epistemic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guided token dropout, which applies the dropout principle to input visual tokens instead of model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding contexts, we can robustly mitigate errors arising from visual token misinterpretations. Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that DROPOUT DECODING significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts. Code is released at https://github.com/kigb/DropoutDecoding.

Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding

TL;DR

This work tackles LVLM hallucinations by introducing Dropout Decoding, a training-free, inference-time strategy that quantifies visual-token uncertainty and masks uncertain tokens. Tokens are projected into the text space to produce per-token distributions, enabling epistemic-uncertainty–driven dropout and ensemble decoding over multiple masked contexts. The approach decomposes uncertainty into aleatoric and epistemic components and focuses on the latter to identify informative visual tokens, with majority voting across K masked decodings yielding more reliable outputs. Empirical results on CHAIR, THRONE, and MMBench demonstrate reduced object hallucinations and improved overall quality across multiple LVLM backbones, with modest computational overhead and scalable efficiency via batching. The method provides a principled, generalizable inference-time technique for enhancing the trustworthiness of multimodal generation without additional training data or fine-tuning.

Abstract

Large vision-language models (LVLMs) excel at multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. We present DROPOUT DECODING, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting it onto the text space and decomposing it into aleatoric and epistemic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guided token dropout, which applies the dropout principle to input visual tokens instead of model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding contexts, we can robustly mitigate errors arising from visual token misinterpretations. Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that DROPOUT DECODING significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts. Code is released at https://github.com/kigb/DropoutDecoding.

Paper Structure

This paper contains 32 sections, 13 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Vision-Language Model Decoding
  5. Uncertainty Quantification
  6. Textual Interpretation of Visual Tokens
  7. Method
  8. Uncertainty Quantification Before Decoding
  9. Uncertainty-Guided Decoding
  10. Experiments
  11. Experimental Setup
  12. CHAIR
  13. THRONE
  14. MMBench
  15. Analysis and Ablation Studies
  16. ...and 17 more sections

Figures (3)

  • Figure 1: An illustrative example where visual tokens are projected into the text space, bold words indicate highly informative projections, and red words mark misalignments. Dotted lines show average uncertainties; high epistemic uncertainty correlates with informative patches.
  • Figure 2: An overview of our Dropout Decoding. The method includes uncertainty measurement of visual tokens (under "Before Decoding") and uncertainty-guided visual context dropout decoding algorithm (under "Decoding Time"). The pseudocode is in Algorithm \ref{['alg:decoding']}.
  • Figure 3: Comparison of CHAIR$_S$, CHAIR$_I$, $P_{all}$ and $R_{all}$ scores with standard deviations across different candidate numbers.