Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artifact-Aware Evaluation for High-Quality Video Generation

Chen Zhu, Jiashu Zhu, Yanxun Li, Meiqi Wu, Bingze Song, Chubin Chen, Jiahong Wu, Xiangxiang Chu, Yangang Wang

TL;DR

This work tackles the challenge of evaluating generated videos with fine-grained artifact localization aligned to human perception. It introduces an artifact-centric protocol based on three perceptual axes (Appearance, Motion, Camera) and a 10-category taxonomy, supported by GenVID, a large-scale annotated dataset, and DVAR, a dense video artifact recognition framework. A Flow-Magnitude–Guided Dynamic Frame Sampling strategy enhances temporal localization, enabling efficient, robust artifact detection when combined with fine-tuned multimodal language models. The results demonstrate improved artifact recognition and effective filtering of low-quality content, offering a scalable solution for moderation and quality control of generated video content.

Abstract

With the rapid advancement of video generation techniques, evaluating and auditing generated videos has become increasingly crucial. Existing approaches typically offer coarse video quality scores, lacking detailed localization and categorization of specific artifacts. In this work, we introduce a comprehensive evaluation protocol focusing on three key aspects affecting human perception: Appearance, Motion, and Camera. We define these axes through a taxonomy of 10 prevalent artifact categories reflecting common generative failures observed in video generation. To enable robust artifact detection and categorization, we introduce GenVID, a large-scale dataset of 80k videos generated by various state-of-the-art video generation models, each carefully annotated for the defined artifact categories. Leveraging GenVID, we develop DVAR, a Dense Video Artifact Recognition framework for fine-grained identification and classification of generative artifacts. Extensive experiments show that our approach significantly improves artifact detection accuracy and enables effective filtering of low-quality content.

Artifact-Aware Evaluation for High-Quality Video Generation

TL;DR

This work tackles the challenge of evaluating generated videos with fine-grained artifact localization aligned to human perception. It introduces an artifact-centric protocol based on three perceptual axes (Appearance, Motion, Camera) and a 10-category taxonomy, supported by GenVID, a large-scale annotated dataset, and DVAR, a dense video artifact recognition framework. A Flow-Magnitude–Guided Dynamic Frame Sampling strategy enhances temporal localization, enabling efficient, robust artifact detection when combined with fine-tuned multimodal language models. The results demonstrate improved artifact recognition and effective filtering of low-quality content, offering a scalable solution for moderation and quality control of generated video content.

Abstract

With the rapid advancement of video generation techniques, evaluating and auditing generated videos has become increasingly crucial. Existing approaches typically offer coarse video quality scores, lacking detailed localization and categorization of specific artifacts. In this work, we introduce a comprehensive evaluation protocol focusing on three key aspects affecting human perception: Appearance, Motion, and Camera. We define these axes through a taxonomy of 10 prevalent artifact categories reflecting common generative failures observed in video generation. To enable robust artifact detection and categorization, we introduce GenVID, a large-scale dataset of 80k videos generated by various state-of-the-art video generation models, each carefully annotated for the defined artifact categories. Leveraging GenVID, we develop DVAR, a Dense Video Artifact Recognition framework for fine-grained identification and classification of generative artifacts. Extensive experiments show that our approach significantly improves artifact detection accuracy and enables effective filtering of low-quality content.
Paper Structure (11 sections, 2 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Artifact-Centric Evaluation Protocol
  4. Construction of GenVID
  5. Flow-Magnitude–Guided Dynamic Frame Sampling
  6. Dense Video Artifact Recognition
  7. Experiments
  8. Experimental Setup
  9. Comparison with SOTA Models
  10. Ablation Studies
  11. Conclusion

Figures (2)

  • Figure 1: Overview of the artifact-aware evaluation pipeline, including GenVID data annotation and the inference workflow of DVAR. Given an input video, FMG-DFS is first employed to sample video clips with a high probability of containing artifacts. These clips are then subjected to inference by a trained MLLM to identify the presence of each potential artifact category.
  • Figure 2: Examples of GenVID dataset. Each generated video containing noticeable artifacts is subjected to human annotation and categorization.