Mitigating Hallucinations in Video Large Language Models via Spatiotemporal-Semantic Contrastive Decoding

TL;DR

VideoLLMs remain prone to hallucinations that conflict with video evidence. The paper introduces Spatiotemporal-Semantic Contrastive Decoding (SSCD), a decoding-time strategy that forms negative video features by disrupting spatiotemporal coherence via a spatiotemporal graph random-walk and weakening semantic alignment through conditional mutual information minimization, then uses these negatives in a calibrated contrastive decoding objective $p_{SSCD}$ to suppress hallucinations. A lightweight Spatiotemporal-Semantic Disruptor (SSD) is trained with a joint loss $\mathcal{L}=\mathcal{L}_{\mathrm{T}}+\lambda\mathcal{L}_{\mathrm{S}}$ and applied during inference with an adaptive plausibility constraint to balance hallucination reduction with linguistic plausibility. Extensive experiments on benchmarks such as VideoHallucer, EventHallusion, and VideoHallu show that SSCD outperforms decoding-based baselines while largely preserving general video understanding and reasoning capabilities, demonstrating a practical, non-intrusive path to more reliable VideoLLMs. The work provides a theoretically grounded decoding framework that explicitly models hallucination-related factors and offers insights into robust temporal reasoning in video-language tasks. Overall, SSCD enables more faithful video-grounded generation without retraining backbone models, with implications for safer and more trustworthy multimodal AI systems.

Abstract

Although Video Large Language Models perform remarkably well across tasks such as video understanding, question answering, and reasoning, they still suffer from the problem of hallucination, which refers to generating outputs that are inconsistent with explicit video content or factual evidence. However, existing decoding methods for mitigating video hallucinations, while considering the spatiotemporal characteristics of videos, mostly rely on heuristic designs. As a result, they fail to precisely capture the root causes of hallucinations and their fine-grained temporal and semantic correlations, leading to limited robustness and generalization in complex scenarios. To more effectively mitigate video hallucinations, we propose a novel decoding strategy termed Spatiotemporal-Semantic Contrastive Decoding. This strategy constructs negative features by deliberately disrupting the spatiotemporal consistency and semantic associations of video features, and suppresses video hallucinations through contrastive decoding against the original video features during inference. Extensive experiments demonstrate that our method not only effectively mitigates the occurrence of hallucinations, but also preserves the general video understanding and reasoning capabilities of the model.

Figures (10)

• Figure 1: Overview of the proposed SSCD. Left: We freeze the VideoLLM and train only a lightweight spatiotemporal-semantic disruptor. Right: During inference, we use the trained disruptor to generate negative video features with disrupted spatiotemporal and semantic consistency, and mitigate hallucinations via calibrated distribution sampling with contrastive decoding.
• Figure 2: Ablation analysis of $\lambda$ in training on the direct multiple-choice subset of MMVU with Video-LLaVA as the backbone.
• Figure 3: Ablation analysis of $\alpha$ and $\beta$ in contrastive decoding on EventHallusion using Video-LLaVA as the backbone.
• Figure 4: An example from the VideoHallu dataset with LLaVA-NeXT-Video as the backbone, where SSCD demonstrates superior temporal information modeling compared to the baseline.
• Figure 5: An example from the MMVU dataset using Video-LLaVA as the backbone. Compared with the baseline, SSCD accurately captures key information across multiple operational steps and correctly interprets and associate it based on domain-specific knowledge.