MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation

TL;DR

MoWM tackles embodied action planning by bridging pixel-space diffusion-based representations with motion-aware latent representations. It train two specialized world models and then modulates low-level pixel features with latent dynamics to form a motion-aware fused representation for action decoding via a diffusion policy. On the CALVIN benchmark, MoWM achieves state-of-the-art task success and demonstrates robust generalization, with ablations confirming the benefit of latent-to-pixel fusion, especially for long-horizon tasks. The approach provides practical guidance on when and how to combine high-level latent dynamics with fine-grained visual details, suggesting strong potential for zero-shot transfer and real-world robotics tasks.

Abstract

Embodied action planning is a core challenge in robotics, requiring models to generate precise actions from visual observations and language instructions. While video generation world models are promising, their reliance on pixel-level reconstruction often introduces visual redundancies that hinder action decoding and generalization. Latent world models offer a compact, motion-aware representation, but overlook the fine-grained details critical for precise manipulation. To overcome these limitations, we propose MoWM, a mixture-of-world-model framework that fuses representations from hybrid world models for embodied action planning. Our approach uses motion-aware representations from a latent model as a high-level prior, which guides the extraction of fine-grained visual features from the pixel space model. This design allows MoWM to highlight the informative visual details needed for action decoding. Extensive evaluations on the CALVIN benchmark demonstrate that our method achieves state-of-the-art task success rates and superior generalization. We also provide a comprehensive analysis of the strengths of each feature space, offering valuable insights for future research in embodied planning. The code is available at: https://github.com/tsinghua-fib-lab/MoWM.

Figures (4)

• Figure 1: Overall framework of MoWM. In the first stage, we independently train a pixel-space and a latent-space world model driven by text and an initial frame. In the second stage, we freeze the world models, perform latent-to-pixel feature modulation, and then end-to-end train an action denoising network for action planning.
• Figure 2: Illustration of the execution process of our model's planned actions in the simulation environment of CALVIN.
• Figure 3: Visualizations of the future state predictions of the latent world model, pixel world model, and the ground truth video.
• Figure 4: Visual feature comparisons during action prediction rollouts. The pixel world model sometimes produces long periods of static frames, lacking dynamic movement. In contrast, the latent world model consistently exhibits better dynamics, demonstrating its strength in learning and predicting motion.