LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation

TL;DR

This work addresses the lack of scalable, multi-dimensional evaluation for AI-generated videos by introducing AIGVE-60K, a large- scale dataset with 58,500 videos, 3,050 prompts across 20 tasks, and 2.6M subjective annotations plus 60K QA pairs. It then presents LOVE, a large multimodal model-based evaluator that jointly handles perceptual quality, text-video correspondence, and task-specific accuracy through dual vision-temporal encoders, an LLM backbone, and instruction tuning with LoRA. Empirical results show LOVE achieves state-of-the-art alignment to human judgments on AIGVE-60K and demonstrates strong zero-shot generalization to other benchmarks, while the dataset enables bidirectional benchmarking of T2V generation and V2T interpretation. The work provides a practical, scalable framework for rigorous AIGV evaluation and highlights areas where current T2V/V2T models struggle, informing future model development and evaluation standards.

Abstract

Recent advancements in large multimodal models (LMMs) have driven substantial progress in both text-to-video (T2V) generation and video-to-text (V2T) interpretation tasks. However, current AI-generated videos (AIGVs) still exhibit limitations in terms of perceptual quality and text-video alignment. Therefore, a reliable and scalable automatic model for AIGV evaluation is desirable, which heavily relies on the scale and quality of human annotations. To this end, we present AIGVE-60K, a comprehensive dataset and benchmark for AI-Generated Video Evaluation, which features (i) comprehensive tasks, encompassing 3,050 extensive prompts across 20 fine-grained task dimensions, (ii) the largest human annotations, including 120K mean-opinion scores (MOSs) and 60K question-answering (QA) pairs annotated on 58,500 videos generated from 30 T2V models, and (iii) bidirectional benchmarking and evaluating for both T2V generation and V2T interpretation capabilities. Based on AIGVE-60K, we propose LOVE, a LMM-based metric for AIGV Evaluation from multiple dimensions including perceptual preference, text-video correspondence, and task-specific accuracy in terms of both instance level and model level. Comprehensive experiments demonstrate that LOVE not only achieves state-of-the-art performance on the AIGVE-60K dataset, but also generalizes effectively to a wide range of other AIGV evaluation benchmarks. These findings highlight the significance of the AIGVE-60K dataset. Database and codes are anonymously available at https://github.com/IntMeGroup/LOVE.

Figures (18)

• Figure 1: We present the largest AIGV evaluation database (AIGVE-60K) and a novel model (LOVE). (a) We collect 3,050 prompts across 20 fine-grained tasks. (b) 30 T2V models are applied to generate 60K videos. (c) 120K MOSs from 2 perspectives and 60K question-answer pairs are acquired from annotators. (d) We design a model to evaluate T2V generation at both the instance and model levels. (e) Comparison results of LMM's V2T interpretation ability based on AIGVE-60K.
• Figure 2: Video examples generated by 30 T2V models using prompt: “a cat is drinking a cup of water”, annotated with MOSs from 2 dimensions: perceptual quality and text-video correpondence.
• Figure 3: (a) Distribution of task counts and averaged scores across different tasks. (b) Distribution of perception and correspondence MOSs.
• Figure 4: The MOS distribution in terms of different task contents. (a) MOS distribution of perception quality (b) MOS distribution of T2V correspondence.
• Figure 5: Comparison of T2V generation models regarding the perception MOSs, correspondence MOSs, and task-specific accuracy.