CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models

TL;DR

The paper tackles hallucinations in video-language models, particularly misidentified actions and incorrect temporal reasoning, by introducing CounterVid, a scalable counterfactual video-generation framework. It leverages multimodal LLMs for action proposals, image editing, and diffusion-based video synthesis to produce semantic hard negatives that preserve scene context. A unified MixDPO objective jointly exploits textual and visual preferences to strengthen grounding and temporal sensitivity. Experiments show that fine-tuning Qwen2.5-VL with MixDPO yields consistent gains on action recognition and temporal ordering, transfers to established hallucination benchmarks, and preserves general video understanding, underscoring the practicality of counterfactual data and preference-based alignment for robust VLMs.

Abstract

Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. Existing mitigation strategies, such as textual filtering or random video perturbations, often fail to address the root cause: over-reliance on language priors rather than fine-grained visual dynamics. We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context. Our pipeline combines multimodal LLMs for action proposal and editing guidance with diffusion-based image and video models to generate semantic hard negatives at scale. Using this framework, we build CounterVid, a synthetic dataset of ~26k preference pairs targeting action recognition and temporal reasoning. We further introduce MixDPO, a unified Direct Preference Optimization approach that jointly leverages textual and visual preferences. Fine-tuning Qwen2.5-VL with MixDPO yields consistent improvements, notably in temporal ordering, and transfers effectively to standard video hallucination benchmarks. Code and models will be made publicly available.

Figures (8)

• Figure 1: Overview of the counterfactual generation framework. Starting from a real video-caption pair, we extract a representative keyframe with a shared embedding space model, propose multiple alternative actions with a multimodal LLM, synthesize one counterfactual video per proposed action via image editing and image-to-video generation, and finally compose preference pairs for action recognition and temporal ordering.
• Figure 2: Qualitative examples from CounterVid, illustrating action recognition and temporal ordering samples across multiple-choice, order-list, binary, and free-form formats. All videos and answers are generated.
• Figure 3: Qualitative examples of hallucination correction on the VidHalluc benchmark. Compared to the base and text-only preference models, MixDPO produces responses that better align with the visual evidence, particularly in cases involving temporal reasoning.
• Figure 4: Action proposal prompt. The VLM is instructed to generate $N$ semantically distinct and physically plausible actions conditioned on the anchor frame, which serve as candidate counterfactuals.
• Figure 5: Action quality evaluation prompt. This prompt filters proposed actions by enforcing subject presence, physical and contextual feasibility, and semantic uniqueness within the scene.