Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion

TL;DR

Envision tackles the challenge of goal-aligned embodied visual planning by explicitly grounding diffusion-based planning in a goal image. It introduces a two-stage approach: a Goal Imagery Model that generates a precise, region-focused goal image, and an Env-Goal Video Model that interpolates a physically plausible trajectory from the start to the goal while conditioning on the goal. The framework employs region-aware cross attention, local edits, and a mixed data training regime to achieve strong cross-domain generalization across human-hand and robot embodiments. Empirical results on object manipulation and image editing benchmarks show improved goal alignment, spatial consistency, and downstream robotic planning performance, underscoring Envision's practical potential for reliable embodied control.

Abstract

Embodied visual planning aims to enable manipulation tasks by imagining how a scene evolves toward a desired goal and using the imagined trajectories to guide actions. Video diffusion models, through their image-to-video generation capability, provide a promising foundation for such visual imagination. However, existing approaches are largely forward predictive, generating trajectories conditioned on the initial observation without explicit goal modeling, thus often leading to spatial drift and goal misalignment. To address these challenges, we propose Envision, a diffusion-based framework that performs visual planning for embodied agents. By explicitly constraining the generation with a goal image, our method enforces physical plausibility and goal consistency throughout the generated trajectory. Specifically, Envision operates in two stages. First, a Goal Imagery Model identifies task-relevant regions, performs region-aware cross attention between the scene and the instruction, and synthesizes a coherent goal image that captures the desired outcome. Then, an Env-Goal Video Model, built upon a first-and-last-frame-conditioned video diffusion model (FL2V), interpolates between the initial observation and the goal image, producing smooth and physically plausible video trajectories that connect the start and goal states. Experiments on object manipulation and image editing benchmarks demonstrate that Envision achieves superior goal alignment, spatial consistency, and object preservation compared to baselines. The resulting visual plans can directly support downstream robotic planning and control, providing reliable guidance for embodied agents.

Figures (9)

• Figure 1: Given a scene image and a task prompt, "Pick up the corn and put it on the right side of sausages," (a) Tesseract zhen2025tesseract performs video generation from a single input image, but struggles to maintain spatial consistency and goal alignment without additional constraints. (b) A strong baseline that combines Nano Banana google2025nanobanana and Wan 2.1 First-Last-Frame-to-Video (FL2V) model wan2025wan: Nano Banana first predicts the goal (last) frame (outlined in blue), and Wan 2.1's FL2V then synthesizes the full sequence. However, Nano Banana often causes unintended scene edits and ambiguous object identities. (c) Our Envision accurately predicts the goal frame (outlined in red) and interpolates between the initial and goal frames, yielding a goal-aligned, spatially consistent, and physically plausible planning video.
• Figure 2: Overview of our Envision framework. Given a single environment image and an instruction prompt, our pipeline generates a physically plausible and goal-aligned video depicting the instructed manipulation in a two-stage manner. Each stage corresponds to a trainable component: (left) a Goal Imagery Model that predicts the target goal frame, and (right) an Env–Goal Video Model that synthesizes the full sequence conditioned on both the environment and goal images.
• Figure 3: Our goal image generation pipeline builds on (a) Flux Kontext batifol2025flux, which interleaves self- and cross-attention between visual and textual tokens. To better guide the model's focus toward regions of interest (ROIs), we extend the input with the corresponding ROI image $I_{roi}$ in addition to the environmental image $I_{env}$, as shown in (b). To further specify where textual instructions apply within $I_{env}$, (c) adds an extra gated cross-attention layer between textual tokens and masked ROIs, termed Local Attention, on top of (b).
• Figure 4: Ablation on our Goal Imagery Model. Without our local mask and local text conditions, global text conditioning tends to misinterpret the instructions. When only the local mask condition is applied, unintended artifacts and objects appear. In contrast, our local mask–text attention helps maintain structural coherence and more precisely follow the edit instructions.
• Figure 5: Comparison of video generation results using goal images produced by Nano Banana google2025nanobanana and Envision. Our method generates goal images that effectively maintain physical consistency and goal alignment, resulting in more coherent and physically plausible video planning sequences.