Inference-time Physics Alignment of Video Generative Models with Latent World Models

TL;DR

This work tackles the failure of video generative models to respect basic physics by reframing physics plausibility as an inference-time alignment problem. It introduces WMReward, which leverages the latent world model VJEPA-2 as a reward signal to tilt sampling and steer denoising trajectories in diffusion-based video generation. Through Best-of-$N$ and gradient-guided schemes, WMReward improves physics plausibility across image-text, image-text-video, and text-video setups, achieving state-of-the-art PhysicsIQ scores and positive human judgments. The study demonstrates the viability of latent world models as physics priors for video generation and discusses scalability, computation tradeoffs, and robustness, pointing to future work on richer rewards and search strategies.

Abstract

State-of-the-art video generative models produce promising visual content yet often violate basic physics principles, limiting their utility. While some attribute this deficiency to insufficient physics understanding from pre-training, we find that the shortfall in physics plausibility also stems from suboptimal inference strategies. We therefore introduce WMReward and treat improving physics plausibility of video generation as an inference-time alignment problem. In particular, we leverage the strong physics prior of a latent world model (here, VJEPA-2) as a reward to search and steer multiple candidate denoising trajectories, enabling scaling test-time compute for better generation performance. Empirically, our approach substantially improves physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings, with validation from human preference study. Notably, in the ICCV 2025 Perception Test PhysicsIQ Challenge, we achieve a final score of 62.64%, winning first place and outperforming the previous state of the art by 7.42%. Our work demonstrates the viability of using latent world models to improve physics plausibility of video generation, beyond this specific instantiation or parameterization.

Figures (10)

• Figure 1: Left: We achieve the new state-of-the-art on PhysicsIQ benchmark, improving physics plausibility in both single (I2V) and multiframe (V2V) conditioned video generation by a considerable margin. Right: Our latent world model reward WMReward substantially outperforms VLM and other vision foundation model-based reward signals under BoN search; the proposed $\nabla$+BoN sampling strategy further improves the scaling effect.
• Figure 2: We improve the physics plausibility of video generation by aligning a pre-trained diffusion model with a latent world model at inference time. Using a reward derived from latent world models, we perform search on guided denoising trajectories to sample from a tilted physics plausible distribution. Compared to the baseline (middle row of each quadrant), our generation (bottom) adheres more closely to real-world physics (top), exhibiting smoother temporal continuity, more accurate solid interactions, and improved fluid behavior.
• Figure 3: Method Overview. We leverage a latent world model, VJEPA-2, to steer video generative models for better physics plausibility. During generation, we apply a sliding window approach and split the generative model's output into sets of context and future frames. We encode generated context frames and predict the embedding of future frames using the latent world model's predictor. Then, we encode the generated future frames and compute the cosine similarity between its embedding and the latent world model prediction, referred to as surprise score. The surprise score serves as reward to search and guide the denoising trajectories.
• Figure 4: Improving Performance via Particle Scaling and Guidance. Visualization of PhysicsIQ score distributions with a Gaussian KDE for MAGI-1 V2V (left) and vLDM I2V (right) generations. Scaling the number of particles (here, 1 vs. 16) yields substantial gains for Best-of-$N$, and adding guidance further sharpens the distribution toward higher physics plausibility. This leads to overall higher average score shown as dashed vertical line.
• Figure 5: Human Study Interface. Annotators view a side-by-side video comparison and indicate their preference on three criteria—Physics Plausibility, Visual Quality, and Prompt Alignment—choosing one of three preference options: Left, Right, or Neutral.