MIND-V: Hierarchical Video Generation for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

TL;DR

To address the data scarcity and error accumulation in long-horizon robotic manipulation video generation, this work introduces MIND-V, a cognition-inspired hierarchical framework that links high-level planning to pixel-level synthesis via SRH, BSB, and MVG. It couples test-time Staged Visual Future Rollouts with a GRPO-based post-training objective using a Physical Foresight Coherence reward to enforce physics-consistent dynamics through a latent world model. The approach achieves state-of-the-art results on long-horizon tasks, demonstrating robust planning, semantic grounding, and physical plausibility across diverse scenarios. This framework offers a scalable, autonomous paradigm for embodied data synthesis with potential impact on imitation learning and sim-to-real transfer.

Abstract

Embodied imitation learning is constrained by the scarcity of diverse, long-horizon robotic manipulation data. Existing video generation models for this domain are limited to synthesizing short clips of simple actions and often rely on manually defined trajectories. To this end, we introduce MIND-V, a hierarchical framework designed to synthesize physically plausible and logically coherent videos of long-horizon robotic manipulation. Inspired by cognitive science, MIND-V bridges high-level reasoning with pixel-level synthesis through three core components: a Semantic Reasoning Hub (SRH) that leverages a pre-trained vision-language model for task planning; a Behavioral Semantic Bridge (BSB) that translates abstract instructions into domain-invariant representations; and a Motor Video Generator (MVG) for conditional video rendering. MIND-V employs Staged Visual Future Rollouts, a test-time optimization strategy to enhance long-horizon robustness. To align the generated videos with physical laws, we introduce a GRPO reinforcement learning post-training phase guided by a novel Physical Foresight Coherence (PFC) reward. PFC leverages the V-JEPA world model to enforce physical plausibility by aligning the predicted and actual dynamic evolutions in the feature space. MIND-V demonstrates state-of-the-art performance in long-horizon robotic manipulation video generation, establishing a scalable and controllable paradigm for embodied data synthesis.

Figures (13)

• Figure 1: Comprehensive comparison of MIND-V against SOTA models for long-horizon robotic video generation.
• Figure 2: Overview of our hierarchical framework for long-horizon robotic manipulation video generation. Beginning in the cognitive core, the Semantic Reasoning Hub (SRH) decomposes a high-level instruction into atomic sub-tasks and plans a detailed trajectory for each. These plans are then encapsulated into our novel Behavioral Semantic Bridge (BSB), a structured, domain-invariant intermediate representation that serves as a precise blueprint for the Motor Video Generator (MVG). The MVG, a conditional diffusion model, renders photorealistic videos that strictly adhere to the kinematic constraints defined in the BSB. At inference time, Staged Visual Future Rollouts provide a "propose-verify-refine” loop for self-correction, ensuring local optimality at each stage to mitigate error accumulation.
• Figure 3: Architecture of the Motor Video Generator (MVG). The MVG utilizes guidance from the BSB to synthesize spatiotemporally precise videos. The process initiates with encoding the BSB's semantic representation into the (c) Spatiotemporal Guidance Tensor, which embeds the visual features of the active agent along its planned trajectory across frames. This tensor is subsequently processed by the (b) Motion Embedding module to produce a refined motion signal ($G$). Finally, this signal is injected into the (a) Latent Diffusion Transformer, conditioning each step of the denoising process to ensure the synthesized video exhibits strict fidelity to the intended motion.
• Figure 4: Physical Foresight Coherence (PFC) Reward. The PFC leverages a frozen V-JEPA2 world model to predict the latent representation of future Target frames conditioned on past Context frames. The reward is the cosine similarity between this prediction and the ground-truth target latent, which measures the video's alignment with the world model's learned physical dynamics.
• Figure 5: Qualitative comparison of long-horizon robotic manipulation video generation. The baseline models exhibit significant deficiencies, including logical inconsistencies, physical implausibility, and poor semantic grounding. In contrast, MIND-V successfully executes long-horizon instructions with high visual quality and physical fidelity. This validates the efficacy of our hierarchical architecture, which decouples high-level reasoning from pixel-level synthesis to ensure robust long-horizon coherence and spatiotemporal precision.