SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Models

TL;DR

SeqBench addresses the gap in text-to-video evaluation by focusing on sequential narrative coherence rather than just frame-level visuals. It introduces a 320-prompt dataset with 2,560 human-annotated videos across eight models, and a Dynamic Temporal Graphs (DTG)–based automatic metric that correlates strongly with human judgments. The framework jointly evaluates visual fidelity and temporal coherence, uncovering critical failure modes such as state inconsistency, multi-object interaction errors, and flawed temporal ordering. The work provides a practical benchmark and measurement approach that can guide future improvements toward robust sequential reasoning in T2V systems and has implications for scalable evaluation in narrative-aware video generation.

Abstract

Text-to-video (T2V) generation models have made significant progress in creating visually appealing videos. However, they struggle with generating coherent sequential narratives that require logical progression through multiple events. Existing T2V benchmarks primarily focus on visual quality metrics but fail to evaluate narrative coherence over extended sequences. To bridge this gap, we present SeqBench, a comprehensive benchmark for evaluating sequential narrative coherence in T2V generation. SeqBench includes a carefully designed dataset of 320 prompts spanning various narrative complexities, with 2,560 human-annotated videos generated from 8 state-of-the-art T2V models. Additionally, we design a Dynamic Temporal Graphs (DTG)-based automatic evaluation metric, which can efficiently capture long-range dependencies and temporal ordering while maintaining computational efficiency. Our DTG-based metric demonstrates a strong correlation with human annotations. Through systematic evaluation using SeqBench, we reveal critical limitations in current T2V models: failure to maintain consistent object states across multi-action sequences, physically implausible results in multi-object scenarios, and difficulties in preserving realistic timing and ordering relationships between sequential actions. SeqBench provides the first systematic framework for evaluating narrative coherence in T2V generation and offers concrete insights for improving sequential reasoning capabilities in future models. Please refer to https://videobench.github.io/SeqBench.github.io/ for more details.

Figures (3)

• Figure 1: Examples of sequential narrative generation failures in current T2V models. Each row demonstrates a specific failure mode: State Consistency (nurse's clothing color changes), Order Sequencing (guard rescues before running), Multi-subject Interaction (wrong subject performs action), and Temporal Dependencies (incorrect liquid color transitions).
• Figure 2: Dataset statistics and prompt characteristics. Left: Word cloud visualization showing the most frequently occurring terms across all 320 prompts in SeqBench, with word size indicating frequency. Right: Distribution of prompt lengths by word count ranges.
• Figure 3: Overview of the Dynamic Temporal Graph (DTG) evaluation framework. The method extracts scene graphs from multiple video frames (left), processes them through the DTG pipeline to generate evaluation questions covering state consistency, action sequencing, and temporal dependencies (center), applies dependency filtering and scoring mechanisms (bottom right), and computes final coherence scores.