Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling

TL;DR

This work introduces Spatiotemporal Skip Guidance (STG), a training-free sampling guidance technique for transformer-based video diffusion models that implicitly creates a weak model by skipping spatiotemporal layers. By aligning the weak and main models through residual and attention skips, along with optional manifold constraints, STG improves video fidelity without sacrificing diversity or motion. The method extends self-perturbation ideas from image diffusion to video by applying perturbations to both spatial and temporal attention, including factorized attention variants. Through experiments on Mochigenmo, Open-Sora, and SVD, STG demonstrates notable gains in imaging quality and realism (FVD, IS, and VBench metrics) and favorable human evaluations, with ablations underscoring the value of both spatial and temporal guidance and later-layer skipping. The approach offers a practical, training-free alternative to CFG and Autoguidance, enabling high-quality video diffusion sampling at scale while highlighting considerations around tuning and potential ethical implications.

Abstract

Diffusion models have emerged as a powerful tool for generating high-quality images, videos, and 3D content. While sampling guidance techniques like CFG improve quality, they reduce diversity and motion. Autoguidance mitigates these issues but demands extra weak model training, limiting its practicality for large-scale models. In this work, we introduce Spatiotemporal Skip Guidance (STG), a simple training-free sampling guidance method for enhancing transformer-based video diffusion models. STG employs an implicit weak model via self-perturbation, avoiding the need for external models or additional training. By selectively skipping spatiotemporal layers, STG produces an aligned, degraded version of the original model to boost sample quality without compromising diversity or dynamic degree. Our contributions include: (1) introducing STG as an efficient, high-performing guidance technique for video diffusion models, (2) eliminating the need for auxiliary models by simulating a weak model through layer skipping, and (3) ensuring quality-enhanced guidance without compromising sample diversity or dynamics unlike CFG. For additional results, visit https://junhahyung.github.io/STGuidance.

Figures (22)

• Figure 1: Visual comparison of video quality between CFG (top row) and our STG method (bottom row). Best viewed in Acrobat Reader; click on the images to watch the videos.
• Figure 2: Comparison between CFG and STG, with the band conceptually representing the noisy data manifold. In STG, the weak model and the main model are aligned along the direction of increasing quality. In contrast, the two models in CFG differ not only in quality but also in aspects such as diversity and prompt alignment capabilities.
• Figure 3: Selected frames from videos generated by Mochi genmo2024mochi with increasing STG scales.
• Figure 4: Comparison of CFG and STG across varying scales in terms of Imaging Quality and FVD.
• Figure 5: Qualitative comparison between CFG and STG on videos generated by Mochi genmo2024mochi.