VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models

TL;DR

VCRBench introduces a video-based long-form causal reasoning benchmark for LVLMs by using shuffled procedural clips to test ordering toward a goal. The study reveals that current LVLMs struggle with integrating multi-step visual events into coherent causal sequences, with human performance far surpassing models. To address this, the authors propose Recognition-Reasoning Decomposition (RRD), a modular two-stage approach that significantly boosts accuracy (up to 25.2 percentage points) by separating video recognition from causal reasoning. The work highlights a heavy reliance on language priors and outlines a clear path for future improvement toward end-to-end visual causal reasoning and more robust evaluation protocols.

Abstract

Despite recent advances in video understanding, the capabilities of Large Video Language Models (LVLMs) to perform video-based causal reasoning remains underexplored, largely due to the absence of relevant and dedicated benchmarks for evaluating causal reasoning in visually grounded and goal-driven settings. To fill this gap, we introduce a novel benchmark named Video-based long-form Causal Reasoning (VCRBench). We create VCRBench using procedural videos of simple everyday activities, where the steps are deliberately shuffled with each clip capturing a key causal event, to test whether LVLMs can identify, reason about, and correctly sequence the events needed to accomplish a specific goal. Moreover, the benchmark is carefully designed to prevent LVLMs from exploiting linguistic shortcuts, as seen in multiple-choice or binary QA formats, while also avoiding the challenges associated with evaluating open-ended QA. Our evaluation of state-of-the-art LVLMs on VCRBench suggests that these models struggle with video-based long-form causal reasoning, primarily due to their difficulty in modeling long-range causal dependencies directly from visual observations. As a simple step toward enabling such capabilities, we propose Recognition-Reasoning Decomposition (RRD), a modular approach that breaks video-based causal reasoning into two sub-tasks of video recognition and causal reasoning. Our experiments on VCRBench show that RRD significantly boosts accuracy on VCRBench, with gains of up to 25.2%. Finally, our thorough analysis reveals interesting insights, for instance, that LVLMs primarily rely on language knowledge for complex video-based long-form causal reasoning tasks.

Figures (12)

• Figure 1: Example question and video. We present an example of video-based long-form causal reasoning task from VCRBench. The correct order is: Clip 1: Cut lemon into slices, Clip 5: Squeeze lemon into the pitcher, Clip 4: Pour lemon juice and water into the pitcher, Clip 3: Stir the lemonade mixture, Clip 2: Pour lemonade into a glass.
• Figure 2: Impact of RRD. Qwen2.5-VL-Instruct$_\text{72B}$ with RRD outperforms Gemini-1.5-Pro and achieve comparable performance to Gemini-2-Flash-Thinking.
• Figure 3: Overview of video construction.Step 1: Given a complete video, key procedural steps are identified based on human-annotated timestamps. Step 2: We keep the key events and discard those that do not depict visual events directly associated with the goal, such as talking or narrating in this example of grilling steak. Step 3: Each key event is shuffled across time and assigned a clip number. These clips are then merged together to form the final test sample.
• Figure 5: Key statistics of our VCRBench.
• Figure 6: Failure examples. Several open-source LVLMs merely list consecutive numbers as the predicted order, exhibiting inability to make a meaningful attempt in VCRBench tasks.