VERHallu: Evaluating and Mitigating Event Relation Hallucination in Video Large Language Models

TL;DR

This work identifies event relation hallucination as a critical blind spot for VideoLLMs and introduces VERHallu, a benchmark targeting causal, temporal, and subevent relations through RC, QA, and CFQA tasks using counterintuitive Mr. Bean videos. The dataset reveals that current VideoLLMs rely on static frame cues and priors, struggling to reason about dense event relations. To address this, the authors propose a training-free Key-Frame Propagating (KFP) strategy that reallocates frame-level attention around informative key frames via a Gaussian propagation mechanism, improving multi-event understanding without hurting inference speed. Empirical results demonstrate substantial reductions in event relation hallucination and better overall reasoning, highlighting a path toward more reliable VideoLLMs in complex temporal contexts.

Abstract

Video Large Language Models (VideoLLMs) exhibit various types of hallucinations. Existing research has primarily focused on hallucinations involving the presence of events, objects, and scenes in videos, while largely neglecting event relation hallucination. In this paper, we introduce a novel benchmark for evaluating the Video Event Relation Hallucination, named VERHallu. This benchmark focuses on causal, temporal, and subevent relations between events, encompassing three types of tasks: relation classification, question answering, and counterfactual question answering, for a comprehensive evaluation of event relation hallucination. Additionally, it features counterintuitive video scenarios that deviate from typical pretraining distributions, with each sample accompanied by human-annotated candidates covering both vision-language and pure language biases. Our analysis reveals that current state-of-the-art VideoLLMs struggle with dense-event relation reasoning, often relying on prior knowledge due to insufficient use of frame-level cues. Although these models demonstrate strong grounding capabilities for key events, they often overlook the surrounding subevents, leading to an incomplete and inaccurate understanding of event relations. To tackle this, we propose a Key-Frame Propagating (KFP) strategy, which reallocates frame-level attention within intermediate layers to enhance multi-event understanding. Experiments show it effectively mitigates the event relation hallucination without affecting inference speed.

Figures (6)

• Figure 1: Comparison of different hallucinations in VideoLLMs. Current video hallucination benchmarks focus on the existence of object, scene, and event in single-event scenarios, while ignoring the event relation hallucination in dense-event scenarios.
• Figure 2: Illustration of VERHallu construction process. V&L bias: vision&language bias. L bias: language bias. GT: ground truth.
• Figure 3: The distribution of video durations in VERHallu.
• Figure 4: Visualization results of the QwenVL-2.5-7B with TAM.
• Figure 5: Vision&Language Bias and Language Bias analysis (%).