Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Construction

TL;DR

This work tackles Aerial Vision-and-Language Navigation (Aerial VLN), where an agent must follow natural language instructions in 3D aerial environments, requiring coordinated vertical and horizontal actions and robust long-horizon reasoning. The authors introduce a grid-based view selection framework that converts action prediction into selecting among six skybox views, integrates a BEV grid map to fuse observations along the path, and employs a cross-modal transformer to align navigation history with the instruction. Key contributions include View-Candidate Correspondence, a BEV local map encoder, vertical action prediction per view, and an extra candidate pool to mitigate occlusions and errors, achieving state-of-the-art results on AerialVLN and AerialVLN-S. The proposed approach demonstrates strong performance gains, improved robustness to occlusions, and flexible compatibility with existing VLN methods, with potential for broader adoption in UAV-guided language navigation tasks.

Abstract

Aerial Vision-and-Language Navigation (Aerial VLN) aims to obtain an unmanned aerial vehicle agent to navigate aerial 3D environments following human instruction. Compared to ground-based VLN, aerial VLN requires the agent to decide the next action in both horizontal and vertical directions based on the first-person view observations. Previous methods struggle to perform well due to the longer navigation path, more complicated 3D scenes, and the neglect of the interplay between vertical and horizontal actions. In this paper, we propose a novel grid-based view selection framework that formulates aerial VLN action prediction as a grid-based view selection task, incorporating vertical action prediction in a manner that accounts for the coupling with horizontal actions, thereby enabling effective altitude adjustments. We further introduce a grid-based bird's eye view map for aerial space to fuse the visual information in the navigation history, provide contextual scene information, and mitigate the impact of obstacles. Finally, a cross-modal transformer is adopted to explicitly align the long navigation history with the instruction. We demonstrate the superiority of our method in extensive experiments.

Figures (3)

• Figure 1: The comparison between aerial VLN and ground-based VLN in discrete environments. Top: The ground-based VLN in discrete environments. The agent receives view candidates at the current position and selects one of them as the next action according to the instruction. Bottom: Aerial VLN. The agent only obtains first-person views and needs to decide both vertical and horizontal actions in continuous environments.
• Figure 2: The overall framework of our method. The agent receives observations skybox at the current position in aerial 3D environments, and the BEV grid map uses the observations to update the cell features. The BEV local map is sent to the BEV map encoder to generate the BEV embedding to provide the context of surrounding environments. The observation encoder encodes the observations along with the extra candidates as candidate embeddings. The agent extracts the instruction embeddings and history embeddings with the corresponding encoder, and the concatenated visual embeddings and word embeddings are sent to the cross-modal encoder for view selection and vertical action. The final prediction is then used to update the extra candidate pool. The figure of aerial 3D environment is from liu2023aerialvln.
• Figure 3: Visualization of a successful navigation of our method. Blue arrows indicate the Forward action; green arrows represent the vertical and turning actions (Turn Left, Turn Right, Ascending, Descending). The final red circle denotes Stop. We highlight aligned landmarks by colored bounding boxes in images and words in the instruction using the same color. The superscript of words denotes the index of the corresponding action or object in images.