Virtual Guidance as a Mid-level Representation for Navigation with Augmented Reality

TL;DR

This work tackles the challenge of conveying abstract navigational instructions to agents operating in dynamic, multi-modal environments. It introduces Virtual Guidance, a visual mid-level representation that overlays non-visual cues onto the agent’s observations, coupled with a sim-to-real framework for transferring policies without fine-tuning. The approach includes simulated rendering of guidance as navigation paths or waypoints, an AR-based real-world rendering pipeline with scene-coordinate regression and PnP pose estimation, and an LLM-assisted language-to-sub-goal translation with open-set object detection. Experimental results in simulation and real-world validation demonstrate that virtual guidance outperforms non-visual baselines and enables robust sim-to-real transfer for AR-augmented navigation tasks.

Abstract

In the context of autonomous navigation, effectively conveying abstract navigational cues to agents in dynamic environments presents significant challenges, particularly when navigation information is derived from diverse modalities such as both vision and high-level language descriptions. To address this issue, we introduce a novel technique termed `Virtual Guidance,' which is designed to visually represent non-visual instructional signals. These visual cues are overlaid onto the agent's camera view and served as comprehensible navigational guidance signals. To validate the concept of virtual guidance, we propose a sim-to-real framework that enables the transfer of the trained policy from simulated environments to real world, ensuring the adaptability of virtual guidance in practical scenarios. We evaluate and compare the proposed method against a non-visual guidance baseline through detailed experiments in simulation. The experimental results demonstrate that the proposed virtual guidance approach outperforms the baseline methods across multiple scenarios and offers clear evidence of its effectiveness in autonomous navigation tasks.

Figures (10)

• Figure 1: Demonstration of (a) the concept of virtual guidance, and (b) the agent’s behaviors in $\{\scalerel*{}{X}, \scalerel*{}{X}, \scalerel*{}{X}\}$ and $\{\scalerel*{}{X}, \scalerel*{}{X}\}$ scenarios.
• Figure 2: The illustration depicts the proposed simulation-to-real transfer framework for navigation tasks with virtual guidance. The color set for rendering is aligned to bridge the domain gaps between simulated and real observations from the DRL agents.
• Figure 3: Left (a)-(e): An overview of different types of virtual guidance schemes compared to the vector based one; Right (f): The illustration of two types of planning schemes considered in the simulated environments for generating virtual guidance.
• Figure 4: The illustrations of sub-goals generated from language instructions with their corresponding virtual guidance. Cases (i) and (ii) demonstrate scenarios with the same object placements but different instructions, while case (iii) presents another example.
• Figure 5: The illustration of the proposed two-stage process to retrieve sub-goal positions through pseudo scene coordinates.