An Efficient and Multi-Modal Navigation System with One-Step World Model

TL;DR

The paper addresses the latency and 3D reasoning limitations of end-to-end and diffusion-based navigation approaches by introducing a one-step, non-autoregressive world model built on a 3D U-Net with spatial-temporal attention. This model simultaneously predicts multiple future frames in latent space, enabling real-time planning when combined with an anchor-based CEM framework that handles multi-modal goals (image, language, point). Key contributions include the one-step generation objective with bootstrap distillation, a computation-efficient architecture, pretraining plus random trajectory sampling to boost data efficiency, and a unified planning approach that demonstrates superior closed-loop performance in simulation and real-world experiments. The findings indicate substantial gains in generation quality, inference speed, and navigation success across multiple modalities, highlighting practical impact for robust, real-time mobile robot navigation.

Abstract

Navigation is a fundamental capability for mobile robots. While the current trend is to use learning-based approaches to replace traditional geometry-based methods, existing end-to-end learning-based policies often struggle with 3D spatial reasoning and lack a comprehensive understanding of physical world dynamics. Integrating world models-which predict future observations conditioned on given actions-with iterative optimization planning offers a promising solution due to their capacity for imagination and flexibility. However, current navigation world models, typically built on pure transformer architectures, often rely on multi-step diffusion processes and autoregressive frame-by-frame generation. These mechanisms result in prohibitive computational latency, rendering real-time deployment impossible. To address this bottleneck, we propose a lightweight navigation world model that adopts a one-step generation paradigm and a 3D U-Net backbone equipped with efficient spatial-temporal attention. This design drastically reduces inference latency, enabling high-frequency control while achieving superior predictive performance. We also integrate this model into an optimization-based planning framework utilizing anchor-based initialization to handle multi-modal goal navigation tasks. Extensive closed-loop experiments in both simulation and real-world environments demonstrate our system's superior efficiency and robustness compared to state-of-the-art baselines.

Figures (7)

• Figure 1: We present a unified navigation framework founded on world models and planning strategies. The proposed system can be directly used to support diverse goal modalities, seamlessly accommodating image-goal, language-goal, and point-goal navigation tasks.
• Figure 2: Illustration of our world model architecture and the world model-based planner. (a) Architecture of the proposed navigation world model. Built upon a 3D U-Net backbone, the model integrates convolutional layers with spatial-temporal attention blocks and is trained using a shortcut objective. (b) The action optimization process conditioned on navigation goals. Leveraging the world model, we utilize a loss function to score action candidates, thereby enabling navigation with multi-modal goals.
• Figure 3: Comparison of anchor-based initialization and random initialization.
• Figure 4: Qualitative comparison of our method against baselines. Given the initial observation, the frames generated by our model achieve higher quality than those of NWM and closely match the ground truth. Furthermore, the visualization demonstrates that random trajectory sampling during training and pretraining on public datasets play a crucial role in performance.
• Figure 5: Visualization of trajectories predicted by different methods in image-goal (left) and language-goal (right) navigation.