VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs

Abstract

Medical Vision-Language Models (VLMs) often hallucinate by generating responses based on language priors rather than visual evidence, posing risks in clinical applications. We propose Visual Grounding Score Guided Decoding (VGS-Decoding), a training-free method to mitigate hallucinations during inference. Our key insight is that hallucinated tokens maintain or increase their probability when visual information is degraded, while visually grounded tokens decrease in probability. We introduce the Visual Grounding Score (VGS), which measures each token's visual dependency by comparing distributions from original and distorted images. During decoding, we reweight probabilities by amplifying visually grounded tokens while suppressing hallucinations. Unlike fixed-weight contrastive methods, VGS-Decoding provides per-token adaptive control. Experiments on MIMIC-Diff-VQA and VQA-RAD across LLaVA-Med, CheXagent, and MedGemma demonstrate consistent improvements, with up to +9.12% overall gain and $+8.98\%$ in open-ended recall, while introducing only $2\times$ inference overhead and no additional training, making it practical for clinical deployment. Upon acceptance, code will be released publicly to facilitate reproducibility.

Figures (2)

• Figure 1: Overview of VGS-Decoding. Both the original image $V$ and its noise-distorted version $V'$ are passed through the medical VLM, producing token distributions $P_{\text{orig}}$ (blue) and $P_{\text{dist}}$ (green). The Visual Grounding Score (VGS) is computed as their normalized probability difference and used for multiplicative reweighting: visually grounded tokens (positive VGS) are amplified, while hallucinated tokens (negative VGS, $\times$) are suppressed.
• Figure 2: Decoding method comparison on VQA-RAD. VGS-Decoding consistently outperforms baselines across all three medical VLMs.