1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)

TL;DR

The paper tackles vision-and-language navigation in continuous environments (VLN-CE) using the RxR-Habitat benchmark.It proposes a modular plan-and-control architecture comprising a Candidate Waypoints Predictor, a History Enhanced Planner, and a Tryout Controller to translate natural-language instructions into low-level actions under no-sliding constraints.The approach leverages in-domain pretraining, environment-level data augmentation, and snapshot ensembling, achieving substantial gains over prior methods on NDTW and SR.On the RxR-Habitat leaderboard, it reaches NDTW 55.43% and SR 45.82%, demonstrating robust performance in unseen environments and offering a strong baseline for future VLN-CE research.

Abstract

This report presents the methods of the winning entry of the RxR-Habitat Competition in CVPR 2022. The competition addresses the problem of Vision-and-Language Navigation in Continuous Environments (VLN-CE), which requires an agent to follow step-by-step natural language instructions to reach a target. We present a modular plan-and-control approach for the task. Our model consists of three modules: the candidate waypoints predictor (CWP), the history enhanced planner and the tryout controller. In each decision loop, CWP first predicts a set of candidate waypoints based on depth observations from multiple views. It can reduce the complexity of the action space and facilitate planning. Then, a history-enhanced planner is adopted to select one of the candidate waypoints as the subgoal. The planner additionally encodes historical memory to track the navigation progress, which is especially effective for long-horizon navigation. Finally, we propose a non-parametric heuristic controller named tryout to execute low-level actions to reach the planned subgoal. It is based on the trial-and-error mechanism which can help the agent to avoid obstacles and escape from getting stuck. All three modules work hierarchically until the agent stops. We further take several recent advances of Vision-and-Language Navigation (VLN) to improve the performance such as pretraining based on large-scale synthetic in-domain dataset, environment-level data augmentation and snapshot model ensemble. Our model won the RxR-Habitat Competition 2022, with 48% and 90% relative improvements over existing methods on NDTW and SR metrics respectively.

Figures (1)

• Figure 1: Overview of our modular plan-and-control scheme.