Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study

Yixiao Jing, Chaoyu Zhang, Zixuan Zhong, Peizhou Huang

TL;DR

This work benchmarks semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts and recommends prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.

Abstract

Semantic noise initialization has been reported to improve robustness and controllability in image diffusion models. Whether these gains transfer to text-to-video (T2V) generation remains unclear, since temporal coupling can introduce extra degrees of freedom and instability. We benchmark semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts. Using prompt-level paired tests with bootstrap confidence intervals and a sign-flip permutation test, we observe a small positive trend on temporal-related dimensions; however, the 95 percent confidence interval includes zero (p ~ 0.17) and the overall score remains on par with the baseline. To understand this outcome, we analyze the induced perturbations in noise space and find patterns consistent with weak or unstable signal. We recommend prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.

Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study

TL;DR

This work benchmarks semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts and recommends prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.

Abstract

Semantic noise initialization has been reported to improve robustness and controllability in image diffusion models. Whether these gains transfer to text-to-video (T2V) generation remains unclear, since temporal coupling can introduce extra degrees of freedom and instability. We benchmark semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts. Using prompt-level paired tests with bootstrap confidence intervals and a sign-flip permutation test, we observe a small positive trend on temporal-related dimensions; however, the 95 percent confidence interval includes zero (p ~ 0.17) and the overall score remains on par with the baseline. To understand this outcome, we analyze the induced perturbations in noise space and find patterns consistent with weak or unstable signal. We recommend prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.
Paper Structure (19 sections, 11 equations, 1 figure, 5 tables)

This paper contains 19 sections, 11 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Experiments
  5. Setup
  6. Quantitative evaluation
  7. Qualitative noise-space diagnostics
  8. Discussion
  9. Conclusion
  10. Additional Qualitative Results
  11. Limitations
  12. Ethics Statement
  13. Reproducibility Statement
  14. Paired significance tests
  15. Additional VBench dimension scores
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Qualitative comparison on VideoCrafter. (Full size view) We visualize samples from Baseline (columns 1, 3, 5) versus our NPNet initialization (columns 2, 4, 6). Our method improves visual fidelity and detail consistency (e.g., the fur of the squirrel and the scales of the lizard) without changing the diffusion backbone.