Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering

TL;DR

This paper addresses hallucination in VideoLLMs and introduces a train-free solution via activation engineering. It uncovers temporal variation as the primary determinant of hallucination sensitivity across internal modules, rather than task type, and presents a temporal-aware framework that automatically constructs task-specific datasets and trains a temporal variation classifier to guide module selection and activation-vector injections. Empirical results on VidHalluc and EventHallusion show consistent, substantial reductions in hallucinations across multiple VideoLLMs without fine-tuning, with quantified gains such as up to +6.15% Overall on VidHalluc over TCD and notable improvements on event-related tasks. The work advances safe, scalable video-language reasoning by enabling robust hallucination mitigation with minimal computational overhead and no additional LLM training, enabling broader deployment in video understanding tasks.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding.However, hallucination, where the model generates plausible yet incorrect outputs, persists as a significant and under-addressed challenge in the video domain. Among existing solutions, activation engineering has proven successful in mitigating hallucinations in LLMs and ImageLLMs, yet its applicability to VideoLLMs remains largely unexplored. In this work, we are the first to systematically investigate the effectiveness and underlying mechanisms of activation engineering for mitigating hallucinations in VideoLLMs. We initially conduct an investigation of the key factors affecting the performance of activation engineering and find that a model's sensitivity to hallucination depends on $\textbf{temporal variation}$ rather than task type. Moreover, selecting appropriate internal modules and dataset for activation engineering is critical for reducing hallucination. Guided by these findings, we propose a temporal-aware activation engineering framework for VideoLLMs, which adaptively identifies and manipulates hallucination-sensitive modules based on the temporal variation characteristic, substantially mitigating hallucinations without additional LLM fine-tuning. Experiments across multiple models and benchmarks demonstrate that our method markedly reduces hallucination in VideoLLMs, thereby validating the robustness of our findings.

Figures (10)

• Figure 1: Overview of two common activation engineering variants.
• Figure 2: Frame reduction amplifies hallucinations in VideoLLM. Rate denotes the frame downsampling rate.
• Figure 3: Binary-classifier performance trained on vector sets $\mathcal{V}_{nh,*}$ from different task types. (b) shows results for $\mathcal{V}_{nh,S}$ with the remaining results presented in Figures \ref{['fig:attn-head-acc-2']} and \ref{['fig:attn-head-acc-3']} of Appendix \ref{['apx:addi_results']}.
• Figure 4: Automated dataset collection pipeline and temporal variation classifier training framework.
• Figure 5: Inference process of the temporal-aware activation engineering framework.