Speaker-Follower Models for Vision-and-Language Navigation

TL;DR

This paper tackles vision-and-language navigation by introducing a speaker–follower framework that reasons about routes and their natural-language descriptions. A panoramic high-level action space is used to simplify planning, while a speaker enables data augmentation and pragmatic inference during both training and testing. The approach yields substantial gains on the Room-to-Room dataset, notably improving unseen-environment success rates and outperforming prior methods with a final 53.5% SR on the test set. These results highlight the value of modeling global navigation structure and pragmatic description-generation in embodied language understanding.

Abstract

Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete low-level motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach---speaker-driven data augmentation, pragmatic reasoning and panoramic action space---dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

Figures (15)

• Figure 1: The task of vision-and-language navigation is to perform a sequence of actions (navigate through the environment) according to human natural language instructions. Our approach consists of an instruction follower model (left) and a speaker model (right).
• Figure 2: Our approach combines an instruction follower model and a speaker model. (a) The speaker model is trained on the ground-truth routes with human-generated descriptions; (b) it provides the follower with additional synthetic instruction data to bootstrap training; (c) it also helps the follower interpret ambiguous instructions and choose the best route during inference. See Sec. \ref{['sec:method']} for details.
• Figure 3: Compared with low-level visuomotor space, our panoramic action space (Sec. \ref{['sec:method_panoramic']}) allows the agents to have a complete perception of the scene, and to directly perform high-level actions.
• Figure 4: Navigation examples on unseen environments with and without pragmatic inference from the speaker model (best visualized in color). (a) The follower without pragmatic inference misinterpreted the instruction and went through a wrong door into a room with no bed. It then stopped at a table (which resembles a bed). (b) With the help of a speaker for pragmatic inference, the follower selected the correct route that enters the right door and stopped at the bed.
• Figure C.1: The average number of actions and navigation error with different speaker weights $\lambda$ in pragmatic inference (Sec. 3.2 of the main paper), evaluated on the val unseen split. Larger $\lambda$ results in more number of actions on average, while $\lambda=0.95$ gives the lowest navigation error.