VideoHEDGE: Entropy-Based Hallucination Detection for Video-VLMs via Semantic Clustering and Spatiotemporal Perturbations

TL;DR

VideoHEDGE addresses the pervasive problem of hallucinations in video-capable vision-language models by introducing a perturbation-aware, semantically grounded reliability framework. It extends the HEDGE approach to temporally structured inputs, using controlled spatiotemporal perturbations and semantic clustering to derive cluster-level reliability scores, of which VASE remains the most effective across models and perturbation budgets. Embedding-based clustering provides near-parallel detection performance to NLI while offering substantial scalability, and domain fine-tuning reduces hallucinations but yields only modest gains in calibration. The framework enables scalable, reproducible evaluation on SoccerChat and highlights practical paths toward robust reliability estimation in Video-VLMs, while also underscoring the need for reliability-aware training to achieve better-calibrated outputs.

Abstract

Hallucinations in video-capable vision-language models (Video-VLMs) remain frequent and high-confidence, while existing uncertainty metrics often fail to align with correctness. We introduce VideoHEDGE, a modular framework for hallucination detection in video question answering that extends entropy-based reliability estimation from images to temporally structured inputs. Given a video-question pair, VideoHEDGE draws a baseline answer and multiple high-temperature generations from both clean clips and photometrically and spatiotemporally perturbed variants, then clusters the resulting textual outputs into semantic hypotheses using either Natural Language Inference (NLI)-based or embedding-based methods. Cluster-level probability masses yield three reliability scores: Semantic Entropy (SE), RadFlag, and Vision-Amplified Semantic Entropy (VASE). We evaluate VideoHEDGE on the SoccerChat benchmark using an LLM-as-a-judge to obtain binary hallucination labels. Across three 7B Video-VLMs (Qwen2-VL, Qwen2.5-VL, and a SoccerChat-finetuned model), VASE consistently achieves the highest ROC-AUC, especially at larger distortion budgets, while SE and RadFlag often operate near chance. We further show that embedding-based clustering matches NLI-based clustering in detection performance at substantially lower computational cost, and that domain fine-tuning reduces hallucination frequency but yields only modest improvements in calibration. The hedge-bench PyPI library enables reproducible and extensible benchmarking, with full code and experimental resources available at https://github.com/Simula/HEDGE#videohedge .

Figures (3)

• Figure 1: Overview of the proposed VideoHEDGE framework for hallucination detection in visual question answering (VQA). A vision--language model generates multiple answers per video--question pair, which are grouped via two strategies: natural language inference (NLI)-based logical clustering and embedding-aligned semantic clustering. Entropy within these groups quantifies uncertainty, enabling hallucination detection through metrics such as RadFlag, Semantic Entropy, and VASE.
• Figure 2: Computational scalability of NLI-based and embedding-based clustering methods for reliability estimation, evaluated using the SoccerChat model on the SoccerChat dataset.
• Figure 3: Effect of sampling scale and the associated visual perturbations on hallucination-detection performance for SoccerChat-qwen2-vl evaluated on the SoccerChat dataset. ROC AUC scores for three metrics (Semantic Entropy, RadFlag, and VASE) evaluated on two tasks: EventClassification and VideoQA. Values reported in Table \ref{['tab:distortion_variations']}.