Self-Refining Video Sampling

TL;DR

Self-Refining Video Sampling addresses the gap between learned priors in diffusion-based video generators and accurate physical dynamics. It reinterprets flow matching as a time-conditioned denoising autoencoder and introduces Predict-and-Perturb (P&P) with an uncertainty-aware gate to refine video latents $z_t$ during inference, without external verifiers or additional training. Across Wan2.1, Wan2.2, and Cosmos-Predict-2.5, the method yields substantial gains in motion coherence, physical realism, and spatial consistency, achieving over 70% human preference over baselines like the default sampler and FlowMo. The approach is practical, training-free, and broadly applicable to improve real-world video generation and visual reasoning tasks.

Abstract

Modern video generators still struggle with complex physical dynamics, often falling short of physical realism. Existing approaches address this using external verifiers or additional training on augmented data, which is computationally expensive and still limited in capturing fine-grained motion. In this work, we present self-refining video sampling, a simple method that uses a pre-trained video generator trained on large-scale datasets as its own self-refiner. By interpreting the generator as a denoising autoencoder, we enable iterative inner-loop refinement at inference time without any external verifier or additional training. We further introduce an uncertainty-aware refinement strategy that selectively refines regions based on self-consistency, which prevents artifacts caused by over-refinement. Experiments on state-of-the-art video generators demonstrate significant improvements in motion coherence and physics alignment, achieving over 70\% human preference compared to the default sampler and guidance-based sampler.

Figures (27)

• Figure 1: Concept of the self-refining video sampling. Within the same noise level, the video latent $z_t$ is refined as the predicted endpoint $\hat{z}_1$ is pulled toward the data manifold.
• Figure 2: Sampling comparison on a 2D synthetic dataset. (a-b) P&P generates samples closer to the data manifold than the Euler solver. (c-d) With a fixed timestep, iterative P&P pulls the prediction $\hat{z}_1$ closer to the data manifold.
• Figure 3: Visualization of uncertainty maps, showing higher values in motion-related regions. Maps are computed at $t = 0.1T$. Bottom row overlays the corresponding binary masks ($\tau=0.25$) on videos generated by Wan2.2-A14B T2V wan2025.
• Figure 4: Qualitative comparison on challenging motion generation.
• Figure 8: Examples of self-refinement applied to visual reasoning tasks: (Top) graph traversal and (Bottom) maze solving from wiedemer2025videomodelszeroshotlearners. We use Wan2.2-A14B I2V as the base model. For graph traversal, self-refinement yields a dramatic improvement in the success rate from 0.1 to 0.8. For maze solving, self-refinement does not yield meaningful gain, with success remaining near zero.