Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NarrativeTrack: Evaluating Video Language Models Beyond the Frame

Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, Gargi Chakraborty

TL;DR

NarrativeTrack addresses the gap in evaluating video-language models for temporally grounded narrative understanding by introducing a fully automated, entity-centric benchmark built around a Compositional Reasoning Progression (CRP) that tests entity existence, changes, and ambiguity. The benchmark couples an automated pipeline for extracting temporally grounded entity trajectories with QA generation across binary, multiple-choice, and ordering formats, enabling fine-grained diagnostics of how models track entities and reason about their evolving states. Experiments show a persistent gap between perceptual grounding and temporal coherence across open-source and proprietary models, with GPT-4o leading but still struggling to maintain robust entity continuity, especially under visual ambiguity or attribute changes. Overall, NarrativeTrack provides a scalable framework to diagnose and advance temporally grounded narrative comprehension in multimodal large language models, highlighting the need for bidirectional temporal modeling and entity-centric training objectives beyond mere model scaling.

Abstract

Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs through fine-grained entity-centric reasoning. Unlike existing benchmarks limited to short clips or coarse scene-level semantics, we decompose videos into constituent entities and examine their continuity via a Compositional Reasoning Progression (CRP), a structured evaluation framework that progressively increases narrative complexity across three dimensions: entity existence, entity changes, and entity ambiguity. CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning. A fully automated entity-centric pipeline enables scalable extraction of temporally grounded entity representations, providing the foundation for CRP. Evaluations of state-of-the-art MLLMs reveal that models fail to robustly track entities across visual transitions and temporal dynamics, often hallucinating identity under context shifts. Open-source general-purpose MLLMs exhibit strong perceptual grounding but weak temporal coherence, while video-specific MLLMs capture temporal context yet hallucinate entity's contexts. These findings uncover a fundamental trade-off between perceptual grounding and temporal reasoning, indicating that narrative understanding emerges only from their integration. NarrativeTrack provides the first systematic framework to diagnose and advance temporally grounded narrative comprehension in MLLMs.

NarrativeTrack: Evaluating Video Language Models Beyond the Frame

TL;DR

NarrativeTrack addresses the gap in evaluating video-language models for temporally grounded narrative understanding by introducing a fully automated, entity-centric benchmark built around a Compositional Reasoning Progression (CRP) that tests entity existence, changes, and ambiguity. The benchmark couples an automated pipeline for extracting temporally grounded entity trajectories with QA generation across binary, multiple-choice, and ordering formats, enabling fine-grained diagnostics of how models track entities and reason about their evolving states. Experiments show a persistent gap between perceptual grounding and temporal coherence across open-source and proprietary models, with GPT-4o leading but still struggling to maintain robust entity continuity, especially under visual ambiguity or attribute changes. Overall, NarrativeTrack provides a scalable framework to diagnose and advance temporally grounded narrative comprehension in multimodal large language models, highlighting the need for bidirectional temporal modeling and entity-centric training objectives beyond mere model scaling.

Abstract

Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs through fine-grained entity-centric reasoning. Unlike existing benchmarks limited to short clips or coarse scene-level semantics, we decompose videos into constituent entities and examine their continuity via a Compositional Reasoning Progression (CRP), a structured evaluation framework that progressively increases narrative complexity across three dimensions: entity existence, entity changes, and entity ambiguity. CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning. A fully automated entity-centric pipeline enables scalable extraction of temporally grounded entity representations, providing the foundation for CRP. Evaluations of state-of-the-art MLLMs reveal that models fail to robustly track entities across visual transitions and temporal dynamics, often hallucinating identity under context shifts. Open-source general-purpose MLLMs exhibit strong perceptual grounding but weak temporal coherence, while video-specific MLLMs capture temporal context yet hallucinate entity's contexts. These findings uncover a fundamental trade-off between perceptual grounding and temporal reasoning, indicating that narrative understanding emerges only from their integration. NarrativeTrack provides the first systematic framework to diagnose and advance temporally grounded narrative comprehension in MLLMs.
Paper Structure (43 sections, 3 equations, 11 figures, 4 tables)

This paper contains 43 sections, 3 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models (MLLMs)
  4. Video Understanding Benchmarks
  5. Evaluating VideoLLMs Beyond the Frame
  6. Problem Statement.
  7. Automated Entity-Centric Pipeline
  8. Entity Detection.
  9. Entity Tracking.
  10. Contextual Recognition.
  11. Pipeline Evaluation.
  12. Compositional Reasoning Progression
  13. QA Generation
  14. Quality Review.
  15. Experiments
  16. ...and 28 more sections

Figures (11)

  • Figure 1: Examples of existing benchmark and NarrativeTrack. While existing benchmarks can often be answered from a single frame, ours requires reasoning by tracking entities over time.
  • Figure 2: Overview of Automated Entity-Centric Pipeline. Our pipeline consists of three key stages: (1) Entity Detection, (2) Entity Tracking, (3) Contextual Recognition, extracting entity representations capturing bounding-box trajectories, actions, outfits, and scene contexts associated with the target entity from raw video without human supervision.
  • Figure 3: Overview of NarrativeTrack. (a) Our benchmark is grounded in Compositional Reasoning Progression that introduces three levels of increasing complexity in entity-centric reasoning: entity existence, entity changes, and entity ambiguity. (b) The benchmark covers three question types: binary, multiple-choice (MC), and ordering, with diverse temporal scales from short to long (average of 55.3 seconds).
  • Figure 4: Temporal Directional Bias of MLLMs. MLLMs encode temporal relations in a forward-only manner and fail to generalize to reversed or bidirectional temporal contexts.
  • Figure 5: Performance Across Track Types. OGP-MLLMs perform best on disappear cases, relying on static visual cues, whereas OVS- and proprietary MLLMs excel on reappear cases, reflecting stronger temporal integration but a higher tendency to hallucinate visual details.
  • ...and 6 more figures