Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation

TL;DR

This work tackles the sim-to-real transfer gap in Vision-and-Language Navigation (VLN) for monocular robots by introducing a Semantic Traversable Map to predict agent-centric traversable regions and candidate waypoints, and 3D Feature Fields to synthesize panoramic, novel-view representations from limited field-of-view inputs. The approach enables monocular robots to emulate near-panoramic perception, allowing high-performance VLN models trained with panoramic data to operate in real-world settings. Extensive experiments in simulation (R2R-CE, RxR-CE) and real-world tests demonstrate improved navigation success rates and obstacle avoidance, with ablations validating the contributions of the semantic map, occupancy map, and 3D feature fields. The proposed framework offers a practical, scalable pathway for deploying high-performing VLN models on common monocular robotic platforms.

Abstract

Vision-and-language navigation (VLN) enables the agent to navigate to a remote location in 3D environments following the natural language instruction. In this field, the agent is usually trained and evaluated in the navigation simulators, lacking effective approaches for sim-to-real transfer. The VLN agents with only a monocular camera exhibit extremely limited performance, while the mainstream VLN models trained with panoramic observation, perform better but are difficult to deploy on most monocular robots. For this case, we propose a sim-to-real transfer approach to endow the monocular robots with panoramic traversability perception and panoramic semantic understanding, thus smoothly transferring the high-performance panoramic VLN models to the common monocular robots. In this work, the semantic traversable map is proposed to predict agent-centric navigable waypoints, and the novel view representations of these navigable waypoints are predicted through the 3D feature fields. These methods broaden the limited field of view of the monocular robots and significantly improve navigation performance in the real world. Our VLN system outperforms previous SOTA monocular VLN methods in R2R-CE and RxR-CE benchmarks within the simulation environments and is also validated in real-world environments, providing a practical and high-performance solution for real-world VLN.

Figures (8)

• Figure 1: The VLN models zhang2024navidchen2022weaklygeorgakis2022cm2 equipped with a monocular camera have limited navigation success rates of less than 39% on the R2R-CE Val Unseen split. Most VLN models an2024etpnavan2023bevbert are trained and evaluated in the simulator 2019habitat with the panoramic observation, achieving navigation success rates of over 57%, but hard to deploy on real-world robots.
• Figure 2: The VLN-CE model an2024etpnav with panoramic RGB-D observation.
• Figure 3: The sim-to-real transfer framework via semantic traversable map and 3D feature fields for vision-and-language navigation.
• Figure 4: The framework of the waypoint predictor with semantic map and occupancy map georgakis2022cm2.
• Figure 5: The arrangement and layout of the real-world lab environment.