Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise

Yunlong Yuan, Yuanfan Guo, Chunwei Wang, Wei Zhang, Hang Xu, Li Zhang

TL;DR

The paper tackles the problem of improving text-to-video diffusion by refining the initial noise prior to mitigate variance decay and detail loss observed in prior methods like FreeInit. It introduces FreqPrior, a three-stage framework with sampling, diffusion, and noise refinement that uses a frequency-domain filter to retain low-frequency structure while enriching high-frequency content, ensuring the refined noise remains close to a standard Gaussian distribution $N(0,I)$. A partial sampling strategy that perturbs the latent at an intermediate timestep further accelerates inference without sacrificing quality. The authors provide a theoretical covariance bound showing the refined noise covariance approaches Gaussian more closely than FreeInit, and validate the approach on VBench across three base models, achieving state-of-the-art quality and semantic scores with about 23% faster inference. Overall, FreqPrior offers a principled, efficient pathway to higher-fidelity, semantically coherent video generation by rethinking the noise prior in the diffusion process.

Abstract

Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, as improved noise priors yield better generation results. One recent approach employs the Fourier transform to manipulate noise, marking the initial exploration of frequency operations in this context. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide a comprehensive theoretical analysis of the variance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain. Our method features a novel filtering technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distribution. Additionally, we propose a partial sampling process by perturbing the latent at an intermediate timestep during finding the noise prior, significantly reducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.

FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise

TL;DR

The paper tackles the problem of improving text-to-video diffusion by refining the initial noise prior to mitigate variance decay and detail loss observed in prior methods like FreeInit. It introduces FreqPrior, a three-stage framework with sampling, diffusion, and noise refinement that uses a frequency-domain filter to retain low-frequency structure while enriching high-frequency content, ensuring the refined noise remains close to a standard Gaussian distribution . A partial sampling strategy that perturbs the latent at an intermediate timestep further accelerates inference without sacrificing quality. The authors provide a theoretical covariance bound showing the refined noise covariance approaches Gaussian more closely than FreeInit, and validate the approach on VBench across three base models, achieving state-of-the-art quality and semantic scores with about 23% faster inference. Overall, FreqPrior offers a principled, efficient pathway to higher-fidelity, semantically coherent video generation by rethinking the noise prior in the diffusion process.

Abstract

Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, as improved noise priors yield better generation results. One recent approach employs the Fourier transform to manipulate noise, marking the initial exploration of frequency operations in this context. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide a comprehensive theoretical analysis of the variance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain. Our method features a novel filtering technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distribution. Additionally, we propose a partial sampling process by perturbing the latent at an intermediate timestep during finding the noise prior, significantly reducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.

Paper Structure

This paper contains 37 sections, 4 theorems, 62 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Video generative models
  4. Noise prior for diffusion models
  5. Method
  6. Diffusion process
  7. Noise refinement
  8. Step 1: Preparation of two sets of noise
  9. Step 2: Retention of low-frequency signals while enriching high-frequency signals
  10. Step 3: Post-processing
  11. Analysis on the distribution of different noise prior
  12. Experiments
  13. Experimental settings
  14. Baselines
  15. Implementations
  16. ...and 22 more sections

Key Result

Lemma A.1

For $\theta=-\frac{2\pi}{N}$ where $N$ is a positive integer, it holds that $\sum_{k=1}^{N}\sin\left(l\left(k-1\right)\theta\right)=0$ for any integer $l$.

Figures (8)

  • Figure 1: (Left)Generated video frames corresponding to Gaussian noise with different variance. As the variance, denoted as $\sigma^2$, decreases from $1.00^2$ to $0.96^2$, the imaging quality deteriorates and background details gradually lost. (Right)Comparisons of our method against the FreeInit and standard Gaussian noise. The frames generated using FreeInit appear overly smooth and blurred in the area of the highlighted red box, whereas our method preserves rich image details.
  • Figure 2: The framework of FreqPrior. It consists of three stages: sampling process, diffusion process, and and noise refinement. In the noise refinement stage, the noise is refined in three steps including noise preparation, noise processing, and post-processing.
  • Figure 3: Generation results using PYoCo prior. Both mixed noise prior and progressive noise prior lead to crashes on pretrained video diffusion models.
  • Figure 4: Qualitative results and comparisons. The cases in the top row are generated using AnimateDiff, while the middle row displays cases from ModelScope, and the bottom row shows cases generated by VideoCrafter. For each case, we present the generation results from different types of noise prior along with the corresponding prompt.
  • Figure 5: Generation results on different values of $\cos\theta$. Though there are some changes in the generated video frames as $\cos\theta$ varies, they are quite similar.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Definition 3.1: Covariance error
  • Lemma A.1
  • proof
  • Theorem A.2
  • proof
  • Theorem A.3
  • Theorem C.1
  • proof