Grounding Generated Videos in Feasible Plans via World Models

TL;DR

Grounding Video Plans with World Models (GVP-WM) tackles the problem that video-generated plans often violate physical constraints and temporal coherence. It grounds such plans at test time by projecting them into dynamically feasible latent trajectories under a pre-trained action-conditioned world model, via video-guided latent collocation and an augmented Lagrangian optimization that balances video guidance, goal attainment, and dynamics. The method executes via model-predictive control, with initialization and alignment losses anchoring the latent trajectory to the video plan, and ALM-based optimization enforcing dynamics. Empirically, GVP-WM recovers feasible long-horizon actions from zero-shot and motion-blurred video plans across navigation and manipulation tasks, outperforming inverse-dynamics baselines and showing robustness to temporal degradation while leveraging domain-adapted video guidance.

Abstract

Large-scale video generative models have shown emerging capabilities as zero-shot visual planners, yet video-generated plans often violate temporal consistency and physical constraints, leading to failures when mapped to executable actions. To address this, we propose Grounding Video Plans with World Models (GVP-WM), a planning method that grounds video-generated plans into feasible action sequences using a learned action-conditioned world model. At test-time, GVP-WM first generates a video plan from initial and goal observations, then projects the video guidance onto the manifold of dynamically feasible latent trajectories via video-guided latent collocation. In particular, we formulate grounding as a goal-conditioned latent-space trajectory optimization problem that jointly optimizes latent states and actions under world-model dynamics, while preserving semantic alignment with the video-generated plan. Empirically, GVP-WM recovers feasible long-horizon plans from zero-shot image-to-video-generated and motion-blurred videos that violate physical constraints, across navigation and manipulation simulation tasks.

Figures (6)

• Figure 1: GVP-WM projects video-generated plans onto dynamically feasible latent trajectories via latent-space trajectory optimization under a pre-trained action-conditioned world model, while preserving semantic alignment with the video plan.
• Figure 2: Overview of GVP-WM. A video plan, which may contain physically infeasible transitions (e.g., motion blur or object teleportation), is encoded into a sequence of latent states ${z^{\mathrm{vid}}_{t:T-1}}$ using a pretrained visual encoder $E\phi$ of the world model. Video-guided latent collocation optimizes a latent trajectory ${z_{t+1:T}}$ and corresponding actions ${a_{t:T-1}}$ by minimizing an augmented Lagrangian objective (Eq. \ref{['eq:auglag']}), which balances video alignment ($L_{\mathrm{vid}}$), terminal goal reaching ($L_{\mathrm{goal}}$), and world-model dynamics ($L_{\mathrm{dyn}}$). The actions from the optimal latent trajectory satisfying the world-model dynamics constraints are executed using model predictive control.
• Figure 3: Qualitative comparison of video-generated plans (top) and GVP-WM executions (bottom). In Push-T, zero-shot video plans (WAN-0S) exhibit physical inconsistencies, including morphological drift (a–b) and rigid-object physics violations (c), while domain-adapted video guidance (WAN-FT) produces physically consistent plans. In Wall, zero-shot video plans exhibit spatial bilocation (e-f).
• Figure 4: The generated video plan (top) hallucinates large artifacts. GVP-WM (bottom) succeeds.
• Figure 7: Zero-shot generated video plan (top) violates kinematics by teleporting the agent to goal state. GVP-WM (bottom) fails.