HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environment

TL;DR

HyPerNav addresses object-oriented navigation in unknown environments by integrating local egocentric RGB-D perception with global top-down-map cues via Vision-Language Models in a training-free framework. It combines three modules—local perception for object cue detection and refinement, global perception for map-guided exploration, and an $A^*$-based planner for motion—along with goal-projection refinement and dilation to improve reachability. Evaluations on HM3D and OVON show state-of-the-art performance in $SR$ and $SPL$, with ablation analyses confirming the contributions of the hybrid perception components. Real-world validation on a lab robot demonstrates efficient navigation and robust performance relative to frontier-based baselines. The work provides a practical, reproducible platform for hybrid perception in embodied navigation and highlights the potential of Vision-Language Models to fuse local and global perceptual cues.

Abstract

Objective-oriented navigation(ObjNav) enables robot to navigate to target object directly and autonomously in an unknown environment. Effective perception in navigation in unknown environment is critical for autonomous robots. While egocentric observations from RGB-D sensors provide abundant local information, real-time top-down maps offer valuable global context for ObjNav. Nevertheless, the majority of existing studies focus on a single source, seldom integrating these two complementary perceptual modalities, despite the fact that humans naturally attend to both. With the rapid advancement of Vision-Language Models(VLMs), we propose Hybrid Perception Navigation (HyPerNav), leveraging VLMs' strong reasoning and vision-language understanding capabilities to jointly perceive both local and global information to enhance the effectiveness and intelligence of navigation in unknown environments. In both massive simulation evaluation and real-world validation, our methods achieved state-of-the-art performance against popular baselines. Benefiting from hybrid perception approach, our method captures richer cues and finds the objects more effectively, by simultaneously leveraging information understanding from egocentric observations and the top-down map. Our ablation study further proved that either of the hybrid perception contributes to the navigation performance.

Figures (7)

• Figure 1: HyPerNav overall process. The left side illustrates the robot explores promising area based on global perception guidance via VLM answers. The right side explains local perception keeps finding the objects along the way until reaching the objectives. The grid map and its corresponding ID is dynamically updated based on explored area. Global perception provide potential target areas area and local perception provides final goal position. The path towards these positions is planned by path planning module based on A star algorithm.
• Figure 2: Refine egocentric goal area with segment anything. Taking "lamp" as an example, directly using the detection bounding box can lead to inaccuracies and unexpected areas after projection onto a top-down map.
• Figure 3: Goal object category frequency distribution in OVON
• Figure 4: In the rightside illustration, the robot would follow global perception guide along path 1 (grey path) if local perception does not find the target bed. As target object is detected by local perception, the robot will stop following path 1 and start following local perception path 2 (pink path) until reaching the object. The ObjNav is then terminated.
• Figure 5: (a) Failures statistics of HM3D. (b) Failures statistics of OVON.