Fine-gained Zero-shot Video Sampling
Dengsheng Chen, Jie Hu, Xiaoming Wei, Enhua Wu
TL;DR
This work introduces Zero-Shot video Sampling (ZS$^2$), a training-free method to extract high-quality, temporally coherent videos from pretrained image diffusion models. It combines a dependency noise model, which imposes KL-divergence guided correlations across frame noises, with temporal momentum attention that blends self- and cross-frame attention to regulate motion while preserving appearance. A two-stage noise search algorithm ensures valid Gaussian noise sequences for long video clips and integrates seamlessly with DDIM sampling, enabling broad compatibility and low overhead. The approach demonstrates strong zero-shot performance, competitive with supervised methods, and enables conditional, specialized, and instruction-guided video generation with minimal additional training, potentially mitigating catastrophic forgetting of image priors. Post-processing with spatio-temporal super-resolution further enhances quality, making ZS$^2$ a practical, scalable pathway for democratizing text-to-video generation from image diffusion priors.
Abstract
Incorporating a temporal dimension into pretrained image diffusion models for video generation is a prevalent approach. However, this method is computationally demanding and necessitates large-scale video datasets. More critically, the heterogeneity between image and video datasets often results in catastrophic forgetting of the image expertise. Recent attempts to directly extract video snippets from image diffusion models have somewhat mitigated these problems. Nevertheless, these methods can only generate brief video clips with simple movements and fail to capture fine-grained motion or non-grid deformation. In this paper, we propose a novel Zero-Shot video Sampling algorithm, denoted as $\mathcal{ZS}^2$, capable of directly sampling high-quality video clips from existing image synthesis methods, such as Stable Diffusion, without any training or optimization. Specifically, $\mathcal{ZS}^2$ utilizes the dependency noise model and temporal momentum attention to ensure content consistency and animation coherence, respectively. This ability enables it to excel in related tasks, such as conditional and context-specialized video generation and instruction-guided video editing. Experimental results demonstrate that $\mathcal{ZS}^2$ achieves state-of-the-art performance in zero-shot video generation, occasionally outperforming recent supervised methods. Homepage: \url{https://densechen.github.io/zss/}.