NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation

TL;DR

This proposed NarrLV is the first benchmark to comprehensively evaluate the Narrative expression capabilities of Long Video generation models and designs an effective evaluation metric using the MLLM-based question generation and answering framework.

Abstract

With the rapid development of foundation video generation technologies, long video generation models have exhibited promising research potential thanks to expanded content creation space. Recent studies reveal that the goal of long video generation tasks is not only to extend video duration but also to accurately express richer narrative content within longer videos. However, due to the lack of evaluation benchmarks specifically designed for long video generation models, the current assessment of these models primarily relies on benchmarks with simple narrative prompts (e.g., VBench). To the best of our knowledge, our proposed NarrLV is the first benchmark to comprehensively evaluate the Narrative expression capabilities of Long Video generation models. Inspired by film narrative theory, (i) we first introduce the basic narrative unit maintaining continuous visual presentation in videos as Temporal Narrative Atom (TNA), and use its count to quantitatively measure narrative richness. Guided by three key film narrative elements influencing TNA changes, we construct an automatic prompt generation pipeline capable of producing evaluation prompts with a flexibly expandable number of TNAs. (ii) Then, based on the three progressive levels of narrative content expression, we design an effective evaluation metric using the MLLM-based question generation and answering framework. (iii) Finally, we conduct extensive evaluations on existing long video generation models and the foundation generation models. Experimental results demonstrate that our metric aligns closely with human judgments. The derived evaluation outcomes reveal the detailed capability boundaries of current video generation models in narrative content expression.

Figures (19)

• Figure 1: (a) Prompt examples with varying numbers of TNAs. (b) Comparison of TNA count distributions across different benchmark.
• Figure 2: Framework of our NarrLV.(a) Our prompt suite is inspired by film narrative theory and identifies three key factors influencing Temporal Narrative Atom (TNA) transitions. Based on these, we construct a prompt generation pipeline capable of producing evaluation prompts with flexibly adjustable TNA counts. (b) Our evaluation models include long video generation models and the foundation models they often rely on. (c) Based on the progressive expression of narrative content, we conduct evaluations from three dimensions, employing an MLLM-based question generation and answering framework for calculations. Our metric is well-aligned with human preferences.
• Figure 2: Comparison of metrics across different benchmarks. Consist-$n$/3 denotes the subset with $n$ consistent results out of three annotations.
• Figure 3: Illustration of our metric evaluation process. Given an evaluation prompt, different video generation models produce corresponding video outputs. Concurrently, based on the semantic information within the prompt, judgment questions concerning different evaluation dimensions are generated, resulting in evaluation outcomes for the generated videos. Better viewed with zoom-in.
• Figure 4: Evaluation results across three evaluation dimensions. Evaluated models include: (a) foundation video generation models and (b) long video generation models.