What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality

TL;DR

This paper proposes a novel metric, VAMP (Visual Appearance and Motion Plausibility), that evaluates both the visual appearance and physical plausibility of generated videos, and demonstrates that VAMP effectively captures both visual fidelity and temporal consistency.

Abstract

As video generation models advance rapidly, assessing the quality of generated videos has become increasingly critical. Existing metrics, such as Fréchet Video Distance (FVD), Inception Score (IS), and ClipSim, measure quality primarily in latent space rather than from a human visual perspective, often overlooking key aspects like appearance and motion consistency to physical laws. In this paper, we propose a novel metric, VAMP (Visual Appearance and Motion Plausibility), that evaluates both the visual appearance and physical plausibility of generated videos. VAMP is composed of two main components: an appearance score, which assesses color, shape, and texture consistency across frames, and a motion score, which evaluates the realism of object movements. We validate VAMP through two experiments: corrupted video evaluation and generated video evaluation. In the corrupted video evaluation, we introduce various types of corruptions into real videos and measure the correlation between corruption severity and VAMP scores. In the generated video evaluation, we use state-of-the-art models to generate videos from carefully designed prompts and compare VAMP's performance to human evaluators' rankings. Our results demonstrate that VAMP effectively captures both visual fidelity and temporal consistency, offering a more comprehensive evaluation of video quality than traditional methods.

Figures (7)

• Figure 1: Comparison of Embedding-Based Metrics and Visual Physics-Based Metrics. The figure illustrates the evaluation pipelines for generated video quality. The top section represents embedding-based metrics, which rely on extracting features from generated videos and comparing them in a latent space against reference videos. The bottom section introduces a human-visual-system-inspired (HVS) evaluation framework, which analyzes videos based on visual appearance (color, shape, texture) and motion coherence (realistic movement dynamics).
• Figure 2: Pipeline for the VAMP Score Calculation. This diagram illustrates the process for calculating the VAMP score. Starting with video input, point sampling and SAM masking are applied to identify and track objects across frames. The pipeline computes the Appearance Score, incorporating metrics for color similarity , shape similarity, and texture similarity. Concurrently, the Motion Score is calculated by evaluating velocity and acceleration consistency, capturing the physical plausibility of object movements. These two scores are combined using weighted factors to compute the final VAMP score.
• Figure 3: Evaluation of video quality using the VAMP metric across different levels of corruptions and generative models. This figure shows the impact of various corruption types on video quality as assessed by the VAMP metric. (Left) Examples of corrupted videos showing the effect of Brightness, Gaussian Noise, Impulse Noise, Defocus Blur, and Black Shapes at different severity levels. (Right) Generated video outputs from state-of-the-art video generation models (T2VZ, Pika, MS, VC2) based on the given prompt.
• Figure 4: Normalized changes in metric values across corruption types and levels. Each heatmap represents the impact of a specific corruption type on four evaluation metrics (ClipSim, IS, FVD, and VAMP). Blue indicates a positive correlation, while red indicates a negative correlation.
• Figure 5: Heatmap of Percentage Changes in VAMP Scores from Level 0 Across Corruption Types Using SIFT Sampling. This figure visualizes the percentage change in VAMP-A (appearance-only), VAMP-M (motion-only), and VAMP (combined) scores across five corruption types: Brightness, Gaussian Noise, Impulse Noise, Defocus Blur, and Black Shapes. Each subplot represents the changes in scores at corruption levels L1 to L5 relative to the baseline at L0.