Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Model

Kwanyoung Kim, Sanghyun Kim

TL;DR

This work tackles the sensitivity of video diffusion outputs to the initial noise seed by introducing ANSE, a model-aware noise selection framework that uses BANSA to quantify attention-based uncertainty. BANSA computes the disagreement in attention maps across stochastic perturbations, selecting seeds with low uncertainty to promote coherent, well-aligned generations. To enable practical deployment, the authors approximate BANSA with Bernoulli-masked attention and identify a small, informative subset of attention layers via cumulative BANSA correlation, achieving near-full BANSA performance with modest inference-time overhead. Across CogVideoX-2B and CogVideoX-5B backbones, ANSE yields improved video quality and semantic alignment on VBench, demonstrating robustness and generalizability of attention-based noise selection for diffusion models. This work introduces a new inference-time scaling principle: improve generation quality by intelligently selecting seeds based on internal model signals rather than retraining or adding heavy priors.

Abstract

The choice of initial noise significantly affects the quality and prompt alignment of video diffusion models, where different noise seeds for the same prompt can lead to drastically different generations. While recent methods rely on externally designed priors such as frequency filters or inter-frame smoothing, they often overlook internal model signals that indicate which noise seeds are inherently preferable. To address this, we propose ANSE (Active Noise Selection for Generation), a model-aware framework that selects high-quality noise seeds by quantifying attention-based uncertainty. At its core is BANSA (Bayesian Active Noise Selection via Attention), an acquisition function that measures entropy disagreement across multiple stochastic attention samples to estimate model confidence and consistency. For efficient inference-time deployment, we introduce a Bernoulli-masked approximation of BANSA that enables score estimation using a single diffusion step and a subset of attention layers. Experiments on CogVideoX-2B and 5B demonstrate that ANSE improves video quality and temporal coherence with only an 8% and 13% increase in inference time, respectively, providing a principled and generalizable approach to noise selection in video diffusion. See our project page: https://anse-project.github.io/anse-project/

Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Model

TL;DR

This work tackles the sensitivity of video diffusion outputs to the initial noise seed by introducing ANSE, a model-aware noise selection framework that uses BANSA to quantify attention-based uncertainty. BANSA computes the disagreement in attention maps across stochastic perturbations, selecting seeds with low uncertainty to promote coherent, well-aligned generations. To enable practical deployment, the authors approximate BANSA with Bernoulli-masked attention and identify a small, informative subset of attention layers via cumulative BANSA correlation, achieving near-full BANSA performance with modest inference-time overhead. Across CogVideoX-2B and CogVideoX-5B backbones, ANSE yields improved video quality and semantic alignment on VBench, demonstrating robustness and generalizability of attention-based noise selection for diffusion models. This work introduces a new inference-time scaling principle: improve generation quality by intelligently selecting seeds based on internal model signals rather than retraining or adding heavy priors.

Abstract

The choice of initial noise significantly affects the quality and prompt alignment of video diffusion models, where different noise seeds for the same prompt can lead to drastically different generations. While recent methods rely on externally designed priors such as frequency filters or inter-frame smoothing, they often overlook internal model signals that indicate which noise seeds are inherently preferable. To address this, we propose ANSE (Active Noise Selection for Generation), a model-aware framework that selects high-quality noise seeds by quantifying attention-based uncertainty. At its core is BANSA (Bayesian Active Noise Selection via Attention), an acquisition function that measures entropy disagreement across multiple stochastic attention samples to estimate model confidence and consistency. For efficient inference-time deployment, we introduce a Bernoulli-masked approximation of BANSA that enables score estimation using a single diffusion step and a subset of attention layers. Experiments on CogVideoX-2B and 5B demonstrate that ANSE improves video quality and temporal coherence with only an 8% and 13% increase in inference time, respectively, providing a principled and generalizable approach to noise selection in video diffusion. See our project page: https://anse-project.github.io/anse-project/

Paper Structure

This paper contains 28 sections, 2 theorems, 20 equations, 10 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Video Diffusion Models
  4. Bayesian Active Learning by Disagreement (BALD)
  5. Methods
  6. BANSA: Bayesian Active Noise Selection via Attention
  7. Stochastic Approximation of BANSA via Bernoulli-Masked Attention
  8. Layer Selection via Cumulative BANSA Correlation
  9. Experiments
  10. Ablation Study and Analysis
  11. Discussion and Limitations
  12. Conclusion
  13. Supplementary Section
  14. Implementation Details of BANSA Score
  15. Score Definition vs. Implementation.
  16. ...and 13 more sections

Key Result

Proposition 1

Let $\mathcal{A}(\mathbf{z}, c, t) = \{A^{(1)}, \dots, A^{(K)}\}$ be a set of row-stochastic attention maps. Then:

Figures (10)

  • Figure 1: Random Seed vs. Ours. We propose ANSE, a noise selection framework, and the BANSA Score, an uncertainty-based metric. By selecting initial noise seeds with lower BANSA scores, which indicate more certain noise samples, ANSE improves video generation performance.
  • Figure 2: Conceptual comparison of noise initialization.(a) Prior methods freeinitfreqprior iteratively refine noise using frequency domain priors through full diffusion sampling, incurring significant computational cost. (b) In contrast, our approach selects optimal noise seeds by estimating attention-based uncertainty at the first denoising step, enabling efficient and model-aware noise selection.
  • Figure 3: Overview of our BANSA-based noise selection process. Given a text prompt $c$, we compute BANSA scores for multiple noise seeds $\{\mathbf{z}_1, \dots, \mathbf{z}_M\}$ using Bernoulli-masked attention maps from selected layers at an early diffusion step. The seed with the lowest score, indicating confident and consistent attention, is selected for generation.
  • Figure 4: Qualitative comparison of CogVideoX variants with and without ANSE. Results from CogVideoX-2B are shown in the first two rows; the rest show results from CogVideoX-5B. With ANSE, videos exhibit improved visual quality, better text alignment, and smoother motion transitions compared to the baseline.
  • Figure 5: Correlation analysis between cumulative BANSA score and full-layer scores. The 0.7 threshold is reached around layer 14 for CogVideoX-2B and layer 19 for CogVideoX-5B.
  • ...and 5 more figures

Theorems & Definitions (5)

  • Definition 1: BANSA Score
  • Proposition 1: BANSA Zero Condition
  • Proposition 1: BANSA Zero Condition
  • proof
  • Remark 1