Direct Motion Models for Assessing Generated Videos

TL;DR

TRAJAN introduces a motion-centric evaluation framework for generated videos by autoencoding dense point tracks into a latent space via a Perceiver-style TRAJectory AutoeNcoder. It enables distribution-level, video-to-video, and per-video assessments, outperforming appearance- and motion-based baselines in sensitivity to temporal distortions and in predicting human judgments of motion realism and consistency. The method also supports spatiotemporal localization of errors through per-point reconstruction quality, enhancing interpretability of video-generation failures. Across diverse datasets and open-source models, TRAJAN demonstrates strong correlations with human ratings and offers a practical, versatile alternative to distribution-focused metrics like FVD when training data or references are limited. Code and results are publicly linked on the project page.

Abstract

A current limitation of video generative video models is that they generate plausible looking frames, but poor motion -- an issue that is not well captured by FVD and other popular methods for evaluating generated videos. Here we go beyond FVD by developing a metric which better measures plausible object interactions and motion. Our novel approach is based on auto-encoding point tracks and yields motion features that can be used to not only compare distributions of videos (as few as one generated and one ground truth, or as many as two datasets), but also for evaluating motion of single videos. We show that using point tracks instead of pixel reconstruction or action recognition features results in a metric which is markedly more sensitive to temporal distortions in synthetic data, and can predict human evaluations of temporal consistency and realism in generated videos obtained from open-source models better than a wide range of alternatives. We also show that by using a point track representation, we can spatiotemporally localize generative video inconsistencies, providing extra interpretability of generated video errors relative to prior work. An overview of the results and link to the code can be found on the project page: http://trajan-paper.github.io.

Figures (17)

• Figure 1: Predicted point trajectories from TRAJAN, colored by their reconstruction error with respect to BootsTAPIR (well reconstructed track, poorly reconstructed track) doersch2024bootstap. See trajan-paper.github.io for videos. The video in (a) has plausible motion but implausible frame-level appearance, while the video in (b) has plausible frame-level appearance but implausible motion (the fingers morph and disappear between frames). TRAJAN correctly predicts this discrepancy by focusing on motion irrespective of appearance.
• Figure 2: TRAJAN -- The trajectory encoder encodes a (variable-sized) set of point trajectories $(x_{t,j},y_{t,j},o_{t,j},t)$ into a compressed motion latent $\phi_S$ of fixed size using a Perceiver jaegle2021perceiver-style transformer architecture. The occlusion flag $o$ is used in the attention mask, making the representation invariant to occluded points. The decoder takes this latent $\phi_S$ and predicts for a query point $(x_q, y_q, t_q)$ the point track that goes through this point at all other times, as well as their occlusion flag. By training the autoencoder on different input and query points, the model learns to represent a dense motion field.
• Figure 3: Synthetic corruptions used to assess sensitivity of evaluation metrics to motion, following ge2024content for the UCF-101 dataset.
• Figure 4: Comparing different methods for detecting temporal distortions on the UCF-101 dataset ge2024content in terms of (left) Fréchet distances in latent space and (right) per-video ordinal scores. TRAJAN and RAFT warp error are particularly sensitive to temporal distortions, while motion histograms liu2024frechet and appearance-based methods perform worse.
• Figure 5: WALT extrapolations for two videos (top: generated, bottom: real).