OSV: One Step is Enough for High-Quality Image to Video Generation

TL;DR

OSV tackles the high computational cost of video diffusion by a two-stage training regimen that blends GAN training with consistency distillation, plus a latent-space discriminator backed by a DINOv2 backbone. A latent upsampling module replaces the VAE decoder, and a high-order Time Travel Sampler enables accurate one-step generation with optional multi-step refinement. Quantitative results on OpenWebVid-1M show state-of-the-art performance for low-step generation, surpassing key baselines and approaching multi-step diffusion quality with far less computation. The approach also presents extensive ablations illustrating the contributions of Stage 1 GAN pretraining, Stage 2 distillation, multi-step solving, and the discriminator design, while noting limitations in complex human motion scenarios.

Abstract

Video diffusion models have shown great potential in generating high-quality videos, making them an increasingly popular focus. However, their inherent iterative nature leads to substantial computational and time costs. While efforts have been made to accelerate video diffusion by reducing inference steps (through techniques like consistency distillation) and GAN training (these approaches often fall short in either performance or training stability). In this work, we introduce a two-stage training framework that effectively combines consistency distillation with GAN training to address these challenges. Additionally, we propose a novel video discriminator design, which eliminates the need for decoding the video latents and improves the final performance. Our model is capable of producing high-quality videos in merely one-step, with the flexibility to perform multi-step refinement for further performance enhancement. Our quantitative evaluation on the OpenWebVid-1M benchmark shows that our model significantly outperforms existing methods. Notably, our 1-step performance(FVD 171.15) exceeds the 8-step performance of the consistency distillation based method, AnimateLCM (FVD 184.79), and approaches the 25-step performance of advanced Stable Video Diffusion (FVD 156.94).

Figures (9)

• Figure 1: OSV is a two-stage video diffusion acceleration strategy. In the first stage, GAN is applied for better training efficiency. In the second stage, we apply consistency distillation to boost the performance upper-bound.
• Figure 2: Summative motivation. We observe and summarize crucial limitations for (latent) consistent models and generalize to the design space, which are well addressed by our approach.
• Figure 3: Comparison of Different Adversarial Training Methods. SF-V requires the encoder of UNet as the feature extraction part of the discriminator. ADD perform adversarial distillation on raw image pixel, which needs to convert latent to image thorough VAE Decoder. In contrast, we directly upsample the latent signal, replacing the decoder with a simple upsampling layer. Only this modification results in a significant speedup in training on NVIDIA H800 at a resolution of 512$\times$512, reducing the average iteration time from 4.29 seconds to 2.61 seconds, and also decreases the occurrence of floating-point overflows during half-precision training. In addition, OSV training consumes 35.8 gigabytes (GB) of GPU memory, a substantial reduction compared to SF-V's 73.5 gigabytes (GB) requirement.
• Figure 4: Overview of OSV. In the first stage, we combine GAN loss and Huber loss song2023improved for better training efficiency. In the second stage, we use consistency distillation loss to boost the performance upper-bound. $b_{sahpe}$, $t_{shape}$, $c_{shape}$, $h_{shape}$ and $w_{shape}$ represent the batch size, number of frames, color channels, height, and width of the input video, respectively.
• Figure 5: Effectiveness of the proposed Time Travel Sampler. Compared to one solver step, multi-step solving exhibits a faster training convergence speed and superior performance, demonstrating the effectiveness of the proposed method. $m$ is set to 5.

Theorems & Definitions (2)

• Theorem A.1
• proof