Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation

TL;DR

Dynam3D tackles VLN in unseen, dynamic 3D environments by introducing dynamic layered 3D representations (patch, instance, zone) that are online-updated as the agent perceives RGB-D data. A dedicated 3D-VLM ingests these tokens, aligned with language through large-scale 3D-language pretraining and targeted distillation, to predict navigation actions in real time. The approach yields state-of-the-art performance on R2R-CE, REVERIE-CE, and NavRAG-CE, with strong gains from pre-exploration and lifelong memory, and demonstrates robust real-world deployment on a mobile robot. The results highlight the value of structured, dynamically updated 3D representations for embodied navigation, enabling improved geometry understanding, long-term memory, and adaptability in changing environments.

Abstract

Vision-and-Language Navigation (VLN) is a core task where embodied agents leverage their spatial mobility to navigate in 3D environments toward designated destinations based on natural language instructions. Recently, video-language large models (Video-VLMs) with strong generalization capabilities and rich commonsense knowledge have shown remarkable performance when applied to VLN tasks. However, these models still encounter the following challenges when applied to real-world 3D navigation: 1) Insufficient understanding of 3D geometry and spatial semantics; 2) Limited capacity for large-scale exploration and long-term environmental memory; 3) Poor adaptability to dynamic and changing environments.To address these limitations, we propose Dynam3D, a dynamic layered 3D representation model that leverages language-aligned, generalizable, and hierarchical 3D representations as visual input to train 3D-VLM in navigation action prediction. Given posed RGB-D images, our Dynam3D projects 2D CLIP features into 3D space and constructs multi-level 3D patch-instance-zone representations for 3D geometric and semantic understanding with a dynamic and layer-wise update strategy. Our Dynam3D is capable of online encoding and localization of 3D instances, and dynamically updates them in changing environments to provide large-scale exploration and long-term memory capabilities for navigation. By leveraging large-scale 3D-language pretraining and task-specific adaptation, our Dynam3D sets new state-of-the-art performance on VLN benchmarks including R2R-CE, REVERIE-CE and NavRAG-CE under monocular settings. Furthermore, experiments for pre-exploration, lifelong memory, and real-world robot validate the effectiveness of practical deployment.

Figures (6)

• Figure 1: Different vision-language large models for monocular VLN tasks. Compared to previous video-based representations (a), our Dynam3D (b) adopts dynamic hierarchical 3D representations offering advantages in spatial geometry and semantic understanding.
• Figure 2: The architecture of our Dynam3D framework. Our Dynam3D takes posed monocular RGB and depth images as input and outputs atomic navigation actions. It encodes and updates multi-level 3D representations for scene understanding and target localization. The 3D tokens, navigation instructions and history actions are then consolidated into the 3D-VLM for next action prediction.
• Figure 3: Left: Illustration of the feature points update and frustum culling strategy. Right: The supervision of feature distillation and 3D-language contrastive learning for our Dynam3D model.
• Figure 4: A demonstration of navigation in a dynamic real-world environment.
• Figure 5: Real-world navigation experiments in static environments.