Aerial Vision-and-Dialog Navigation

TL;DR

<3-5 sentence high-level summary> This paper presents Aerial Vision-and-Dialog Navigation (AVDN), introducing a photorealistic drone simulator and a dataset of 3,064 aerial navigation trajectories collected via asynchronous commander–follower dialogs with human attention annotations. It defines two navigation tasks, ANDH and ANDH-Full, that require grounding natural language in continuous aerial imagery. The authors propose the HAA-Transformer, a multimodal transformer that jointly predicts navigation waypoints and human attention, showing that attention-grounded, multimodal learning improves navigation performance, especially on longer trajectories. The work enables hands-free, dialog-guided aerial control and provides a foundation for future vision-and-language navigation research in aerial domains.

Abstract

The ability to converse with humans and follow natural language commands is crucial for intelligent unmanned aerial vehicles (a.k.a. drones). It can relieve people's burden of holding a controller all the time, allow multitasking, and make drone control more accessible for people with disabilities or with their hands occupied. To this end, we introduce Aerial Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language conversation. We build a drone simulator with a continuous photorealistic environment and collect a new AVDN dataset of over 3k recorded navigation trajectories with asynchronous human-human dialogs between commanders and followers. The commander provides initial navigation instruction and further guidance by request, while the follower navigates the drone in the simulator and asks questions when needed. During data collection, followers' attention on the drone's visual observation is also recorded. Based on the AVDN dataset, we study the tasks of aerial navigation from (full) dialog history and propose an effective Human Attention Aided Transformer model (HAA-Transformer), which learns to predict both navigation waypoints and human attention.

Figures (14)

• Figure 1: An example of Aerial Vision-and-Dialog Navigation (AVDN). The user instructs the agent to fly to a destination. During the navigation, the agent can ask questions while showing the images of past visual observations and relative trajectory. The user will talk back at a convenient time to provide further guidance to the agent without having to monitor the agent all the time.
• Figure 2: Example of a trajectory in our AVDN dataset. On the left, the commander's turn and the follower's turn alternate in chronological order. In each turn, dialog utterances are shown, and the follower's turn also shows the navigation process that spans from time step $T$ to $T+1$, including the follower's observation and attention. On the right, there are trajectory overviews at different time steps. More examples can be found in the Appendix.
• Figure 3: (a) displays the frequent words that appear in the dialogs and (b) shows the path length distribution of our AVDN dataset.
• Figure 4: Our Human Attention Aided (HAA) model. The output of the model will interact with our simulator for generating the input for next time step.
• Figure 5: The impact of human attention prediction training on the success of trajectories of different lengths. Human attention prediction significantly improves navigation performance for longer trajectories.