FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation

TL;DR

The paper introduces FETV, a Fine-grained Evaluation Benchmark for open-domain text-to-video generation, addressing the need for temporally-aware, multi-aspect evaluation beyond coarse overall metrics. It defines a three-axis categorization (major content, attribute control, prompt complexity) plus temporal categories, and collects 619 prompts with reference videos to support manual evaluation of four open T2V models. By diagnosing automatic metrics, it reveals weak overall alignment with human judgments for FID, FVD, and CLIPScore, and proposes two UMT-based metrics (FVD-UMT and UMTScore) that better correlate with human outcomes. The results highlight specific weaknesses in current models for temporal content and controllability, and demonstrate that FETV can serve as a diagnostic tool for metric reliability, guiding future improvements in T2V evaluation. Overall, FETV provides a practical, interpretable framework to compare models and validate metrics in open-domain T2V generation with potential for broad adoption and extension.

Abstract

Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.

Figures (31)

• Figure 1: Illustration of our multi-aspect categorization (a-b), based on which we can realize fine-grained evaluation (c). Details of each category are shown in Figure \ref{['fig:temporal_content_attribute']} and Appendix \ref{['sec:category_describe_app']}.
• Figure 2: Descriptions and examples of the temporal categories.
• Figure 3: Data distribution over categories under the "major content" (upper) and "attribute control" (lower) aspects.
• Figure 5: Manual evaluation of static and temporal video quality.
• Figure 6: Manual evaluation of video-text alignment.