Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models

Aaron Foss, Chloe Evans, Sasha Mitts, Koustuv Sinha, Ammar Rizvi, Justine T. Kao

TL;DR

CausalVQA introduces a real-world video question-answering benchmark focused on physical causal reasoning, spanning five question types (counterfactual, hypothetical, anticipation, planning, descriptive) and built on egocentric EgoExo4D videos. The dataset employs a rigorous, multi-stage human-AI generation pipeline to ensure visual grounding and reduce linguistic shortcuts, including distractor refinement and language perturbation. Empirical results show a substantial gap between humans and state-of-the-art multimodal models, particularly on anticipation and hypothetical reasoning, underscoring the need for improved spatial-temporal and physical-commonsense reasoning. By providing paired/unpaired evaluation and a large human baseline, CausalVQA aims to drive progress toward models that truly understand real-world dynamics and causality rather than relying on surface cues.

Abstract

We introduce CausalVQA, a benchmark dataset for video question answering (VQA) composed of question-answer pairs that probe models' understanding of causality in the physical world. Existing VQA benchmarks either tend to focus on surface perceptual understanding of real-world videos, or on narrow physical reasoning questions created using simulation environments. CausalVQA fills an important gap by presenting challenging questions that are grounded in real-world scenarios, while focusing on models' ability to predict the likely outcomes of different actions and events through five question types: counterfactual, hypothetical, anticipation, planning and descriptive. We designed quality control mechanisms that prevent models from exploiting trivial shortcuts, requiring models to base their answers on deep visual understanding instead of linguistic cues. We find that current frontier multimodal models fall substantially below human performance on the benchmark, especially on anticipation and hypothetical questions. This highlights a challenge for current systems to leverage spatial-temporal reasoning, understanding of physical principles, and comprehension of possible alternatives to make accurate predictions in real-world settings.

CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models

TL;DR

CausalVQA introduces a real-world video question-answering benchmark focused on physical causal reasoning, spanning five question types (counterfactual, hypothetical, anticipation, planning, descriptive) and built on egocentric EgoExo4D videos. The dataset employs a rigorous, multi-stage human-AI generation pipeline to ensure visual grounding and reduce linguistic shortcuts, including distractor refinement and language perturbation. Empirical results show a substantial gap between humans and state-of-the-art multimodal models, particularly on anticipation and hypothetical reasoning, underscoring the need for improved spatial-temporal and physical-commonsense reasoning. By providing paired/unpaired evaluation and a large human baseline, CausalVQA aims to drive progress toward models that truly understand real-world dynamics and causality rather than relying on surface cues.

Abstract

We introduce CausalVQA, a benchmark dataset for video question answering (VQA) composed of question-answer pairs that probe models' understanding of causality in the physical world. Existing VQA benchmarks either tend to focus on surface perceptual understanding of real-world videos, or on narrow physical reasoning questions created using simulation environments. CausalVQA fills an important gap by presenting challenging questions that are grounded in real-world scenarios, while focusing on models' ability to predict the likely outcomes of different actions and events through five question types: counterfactual, hypothetical, anticipation, planning and descriptive. We designed quality control mechanisms that prevent models from exploiting trivial shortcuts, requiring models to base their answers on deep visual understanding instead of linguistic cues. We find that current frontier multimodal models fall substantially below human performance on the benchmark, especially on anticipation and hypothetical questions. This highlights a challenge for current systems to leverage spatial-temporal reasoning, understanding of physical principles, and comprehension of possible alternatives to make accurate predictions in real-world settings.

Paper Structure

This paper contains 41 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. The CausalVQA Dataset
  3. Design Objectives
  4. Generation Process
  5. Video Selection
  6. Initial Question Generation
  7. Distractor Generation
  8. Language Refinement
  9. Language Perturbation and Reordering
  10. Ensuring Visual Groundedness
  11. Human Quality Assurance
  12. Human-Derived Empirical Difficulty Levels
  13. Dataset Description
  14. Benchmark Composition
  15. Debug Dataset
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Process of generating and curating question-answer pairs for CausalVQA. Multiple steps were incorporated in order to ensure diversity and visual groundedness as well as reduce susceptibility to shortcuts.
  • Figure 2: Breakdowns of question pairs by question category and difficulty level.
  • Figure 3: Videos in each question type and difficulty level have generally similar and overlapping duration distributions.