Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuning

TL;DR

This work targets temporal inconsistency in image-to-video generation by introducing Video Consistency Distance (VCD), a frequency-domain reward that ties generated frames to the conditioning image. VCD compares amplitude and phase distributions of frame features, computed through a Wasserstein distance in the frequency domain, and is incorporated into a reward-based fine-tuning framework for diffusion models. Experimental results across multiple datasets and two strong baselines show that VCD improves temporal coherence while preserving visual quality and alignment, with robustness on unseen artistic domains. The approach is memory-efficient (e.g., uses LoRA) and does not require real-world video data, offering practical gains for I2V applications.

Abstract

Reward-based fine-tuning of video diffusion models is an effective approach to improve the quality of generated videos, as it can fine-tune models without requiring real-world video datasets. However, it can sometimes be limited to specific performances because conventional reward functions are mainly aimed at enhancing the quality across the whole generated video sequence, such as aesthetic appeal and overall consistency. Notably, the temporal consistency of the generated video often suffers when applying previous approaches to image-to-video (I2V) generation tasks. To address this limitation, we propose Video Consistency Distance (VCD), a novel metric designed to enhance temporal consistency, and fine-tune a model with the reward-based fine-tuning framework. To achieve coherent temporal consistency relative to a conditioning image, VCD is defined in the frequency space of video frame features to capture frame information effectively through frequency-domain analysis. Experimental results across multiple I2V datasets demonstrate that fine-tuning a video generation model with VCD significantly enhances temporal consistency without degrading other performance compared to the previous method.

Figures (15)

• Figure 1: The examples that previous methods opensoraprabhudesai2024videodiffusionalignmentreward fail to preserve temporal consistency in I2V generation. On the top, we compare the pre-trained Open-Sora opensora with its generated by a model fine-tuned with VCD (+VCD). Open-Sora shows a significant collapse in the last frame. In the middle and bottom, we compare VADER, where two kinds of reward functions are employed, and our approach. In the middle, we present videos generated by Open-Sora models fine-tuned with two reward functions: V-JEPA bardes2024revisiting (+V-JEPA) and VCD. +V-JEPA significantly alters the various attributes of the conditioning image. On the bottom, we present videos generated by Open-Sora models fine-tuned with the LAION Aesthetic predictor aesthetic (+Aesthetic) and VCD. +Aeshtetic significantly alters the style of the conditioning image. In contrast to previous methods, ours generates temporally consistent videos relative to the conditioning images.
• Figure 2: The overview of the proposed method. Given the generated video $X$, we randomly sample $x_i$ to calculate Video Consistency Distance (VCD) between the conditioning image $x_{\mathrm{cnd}}$ and $x_i$. VCD utilizes Discrete Fourier Transform (DFT)-based frequency component extraction and temporal weighting to preserve attributes of the conditioning image $x_{\mathrm{cnd}}$ while providing appropriate motion.
• Figure 3: Results of video generation with I2V-Bench. We provide text prompts below the figures.
• Figure 4: Results of video generation with VBench-I2V.
• Figure 5: Results of video generation with AI-ArtBench.

Theorems & Definitions (2)

• Remark 1
• Remark 2