Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models

TL;DR

This work proposes a simple latent processing method that dramatically reduces memory usage, and applies a novel latent optimization strategy designed for globally coherent video generation, called Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals.

Abstract

Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.

Figures (30)

• Figure 1: Frame Guidance enables training-free controllable video generation using flexible frame-level inputs. It supports diverse applications, including keyframe-guided generation, stylization, and looping, using general frame-level inputs such as depth maps, sketches, and color blocks.
• Figure 2: Frame Guidance steers the video generation process of a VDM by applying gradient-based guidance to selected frames, resulting in a temporally coherent controlled video. Our method is training-free, model-agnostic, and supports a wide range of frame-level conditions.
• Figure 3: Frame Guidance for keyframe-guided video generation task. (Left) Illustration of our method with latent slicing and spatial down-sampling (\ref{['method:latent-slicing']}), and gradient propagation with L2 loss (red arrows; \ref{['method:frame-guidance']}). (Right) Visualization of the video latent optimization (VLO; \ref{['method:VLO']}), showing the generated video frames at each guided inference step.
• Figure 4: (a)GPU memory for guidance when using full latent sequence, sliced latents, and latent slicing with spatial down-sampling. (b)Temporal locality of CausalVAEs. Each latent (y-axis) is primarily affected by a small subset of temporally local video frames. (c)Guidance influence during the denoising steps. Yellow arrows indicate the location for the guidance frame.
• Figure 5: Qualitative comparison on keyframe-guided video generation tasks. Yellow box indicates the keyframe condition. Orange box in (a) shows a disconnection in SVD-Interp. Red box in (d) visualizes a failure case for the CogX-Interp baseline for dynamic human motion.