Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Seunghwan Bang, Hwanjun Song

Abstract

The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extractive reasoning, where answers can be explicitly presented within spatiotemporal events. It remains unclear whether multimodal large language models can instead perform abstractive spatiotemporal reasoning, which requires integrating observations over time, combining dispersed cues, and inferring implicit spatial and contextual structure. To address this gap, we formalize abstractive spatiotemporal reasoning from videos by introducing a structured evaluation taxonomy that systematically targets its core dimensions and construct a controllable, scenario-driven synthetic egocentric video dataset tailored to evaluate abstractive spatiotemporal reasoning capabilities, spanning object-, room-, and floor-plan-level scenarios. Based on this framework, we present VAEX-BENCH, a benchmark comprising five abstractive reasoning tasks together with their extractive counterparts. Our extensive experiments compare the performance of state-of-the-art MLLMs under extractive and abstractive settings, exposing their limitations on abstractive tasks and providing a fine-grained analysis of the underlying bottlenecks. The dataset will be released soon.

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Abstract

The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extractive reasoning, where answers can be explicitly presented within spatiotemporal events. It remains unclear whether multimodal large language models can instead perform abstractive spatiotemporal reasoning, which requires integrating observations over time, combining dispersed cues, and inferring implicit spatial and contextual structure. To address this gap, we formalize abstractive spatiotemporal reasoning from videos by introducing a structured evaluation taxonomy that systematically targets its core dimensions and construct a controllable, scenario-driven synthetic egocentric video dataset tailored to evaluate abstractive spatiotemporal reasoning capabilities, spanning object-, room-, and floor-plan-level scenarios. Based on this framework, we present VAEX-BENCH, a benchmark comprising five abstractive reasoning tasks together with their extractive counterparts. Our extensive experiments compare the performance of state-of-the-art MLLMs under extractive and abstractive settings, exposing their limitations on abstractive tasks and providing a fine-grained analysis of the underlying bottlenecks. The dataset will be released soon.
Paper Structure (26 sections, 7 figures, 14 tables)

This paper contains 26 sections, 7 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Taxonomy of Video Spatiotemporal Reasoning
  4. Expansion from Extractive to Abstractive Tasks
  5. VAEX-Bench Dataset
  6. Data Construction Pipeline
  7. Dataset Overview and Human Process
  8. Evaluation
  9. Main Result: Abstractive vs. Extractive Reasoning
  10. From MCQ to Free-form Generation
  11. Detailed Analysis & Discussion
  12. Conclusion
  13. Details of Dataset Construction
  14. Procedural Synthesis of Environments
  15. Human-in-the-Loop Annotation
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Two examples of abstractive spatiotemporal queries from VAEX-Bench: (Left) Global Counting, requiring aggregation of objects observed across the traversal; (Right) Map Direction, requiring inference of relative spatial directions between rooms.
  • Figure 2: Query-conditioned video construction pipeline: (Step 1) Scenario-based Query Construction, followed by (Step 2) Environment and Trajectory Design and (Step 3) Rendering and Video Recording.
  • Figure 3: Error breakdown for the Global Counting task.
  • Figure 4: Performance on VAEX-Bench in free-form generation, where the MCQ-format questions of the abstractive tasks are converted into open-ended queries without providing answer choices.
  • Figure 4: Temporal bottleneck in multi-hop sub-tasks. The outer circle represents the pass/fail ratio for Hop 1 (Blue: Pass, Pink: Fail). The inner ring shows the conditional pass rate for Hop 2 among the cases that passed Hop 1 (Green: Pass, Yellow: Fail).
  • ...and 2 more figures