RANGER: A Monocular Zero-Shot Semantic Navigation Framework through Contextual Adaptation

TL;DR

RANGER addresses the challenge of zero-shot semantic navigation using only monocular RGB input, eliminating the need for depth sensing or precise pose estimation. It introduces a keyframe-based memory bank that fuses geometric and semantic information and leverages a VLM-guided, hierarchical planner to adapt navigation strategies, including rapid adaptation from short offline videos. The approach achieves competitive performance on HM3D without depth data and demonstrates superior context learning in real-world tests, reducing exploration through contextual priors. This work offers a practical path toward real-world RGB-only autonomous navigation and emphasizes the value of in-context learning for rapid environmental adaptation.

Abstract

Efficiently finding targets in complex environments is fundamental to real-world embodied applications. While recent advances in multimodal foundation models have enabled zero-shot object goal navigation, allowing robots to search for arbitrary objects without fine-tuning, existing methods face two key limitations: (1) heavy reliance on precise depth and pose information provided by simulators, which restricts applicability in real-world scenarios; and (2) lack of in-context learning (ICL) capability, making it difficult to quickly adapt to new environments, as in leveraging short videos. To address these challenges, we propose RANGER, a novel zero-shot, open-vocabulary semantic navigation framework that operates using only a monocular camera. Leveraging powerful 3D foundation models, RANGER eliminates the dependency on depth and pose while exhibiting strong ICL capability. By simply observing a short video of a new environment, the system can also significantly improve task efficiency without requiring architectural modifications or fine-tuning. The framework integrates several key components: keyframe-based 3D reconstruction, semantic point cloud generation, vision-language model (VLM)-driven exploration value estimation, high-level adaptive waypoint selection, and low-level action execution. Experiments on the HM3D benchmark and real-world environments demonstrate that RANGER achieves competitive performance in terms of navigation success rate and exploration efficiency, while showing superior ICL adaptability, with no previous 3D mapping of the environment required.

Figures (6)

• Figure 1: RANGER Workflow: Given an offline video of a new environment captured with a camera (optionally) and an online RGB observation stream from the robot, RANGER efficiently adapts to the new environment and completes navigation tasks based on human instructions.
• Figure 2: Overview of RANGER: a unified architecture for zero-shot RGB-only navigation that builds geometric and semantic memory via a keyframe-based memory bank, which subsequently guides the agent’s exploration.
• Figure 3: Visualization of the navigation process in a simulated environment with task instruction: "Find the toilet". (a) Reconstructed 3D map and the agent's trajectory at step 127. The red cones represent keyframe locations, while the green cone indicates the agent's current position. The green line traces the agent's movement, and the red point cloud represents the toilet object. (b) The Frontier Map showing the robot's exploration trajectory. The blue area highlights the target object region, red areas indicate obstacles, and yellow dots represent frontier points. (c) The Value Map, where red regions correspond to high-value areas. (d) First-person observation.
• Figure 4: Contextual adaptation for zero-shot navigation in a new environment. RANGER can leverage an offline video to adapt and locate the target object "bed". (a-d) Initial state with offline memory; (e-h) Final navigation state. The system successfully found the target in 38 steps.
• Figure 5: Comparison of 3D maps reconstructed using different methods. Figure (a) shows the result using real depth and pose data. Figures (b), (c), and (d) present the reconstruction results of VGGT, VGGT-SLAM, and MASt3R-SLAM, respectively, all relying solely on RGB images.