Temporal-Spatial Object Relations Modeling for Vision-and-Language Navigation

TL;DR

This work addresses Vision-and-Language Navigation by modeling object relations along time and across space using Temporal Object Relations (TOR) and Spatial Object Relations (SOR), integrated into a DUET-based VLN framework. It introduces a Turning Back Penalty (TBP) loss to discourage redundant revisits, improving path efficiency. The method is pretrained with MLM, MRC, SAP, and OG tasks and fine-tuned with a pseudo-interactive demonstrator, achieving superior results on REVERIE and SOON and competitive results on R2R. The combination of temporal/spatial object relations and TBP yields more accurate navigation decisions and shorter, more direct trajectories, with qualitative analyses illustrating meaningful attention to object interactions. The approach advances object-relations modeling in VLN and demonstrates practical gains in navigation performance.

Abstract

Vision-and-Language Navigation (VLN) is a challenging task where an agent is required to navigate to a natural language described location via vision observations. The navigation abilities of the agent can be enhanced by the relations between objects, which are usually learned using internal objects or external datasets. The relationships between internal objects are modeled employing graph convolutional network (GCN) in traditional studies. However, GCN tends to be shallow, limiting its modeling ability. To address this issue, we utilize a cross attention mechanism to learn the connections between objects over a trajectory, which takes temporal continuity into account, termed as Temporal Object Relations (TOR). The external datasets have a gap with the navigation environment, leading to inaccurate modeling of relations. To avoid this problem, we construct object connections based on observations from all viewpoints in the navigational environment, which ensures complete spatial coverage and eliminates the gap, called Spatial Object Relations (SOR). Additionally, we observe that agents may repeatedly visit the same location during navigation, significantly hindering their performance. For resolving this matter, we introduce the Turning Back Penalty (TBP) loss function, which penalizes the agent's repetitive visiting behavior, substantially reducing the navigational distance. Experimental results on the REVERIE, SOON, and R2R datasets demonstrate the effectiveness of the proposed method.

Figures (5)

• Figure 1: Three methods of learning the connections between objects
• Figure 2: The overall network architecture. (a) The baseline utilizes a dual-scale encoder to encode local panoramic features, global historical features, and instruction features for action prediction of the agent. (b) At each time step $t$, our method employ two modules to learn temporal and spatial object relations. Then the object relation features are combined with the image features for action prediction. Finally, we designs a novel TBP loss function to supervise the training of the agent in order to reduce its tendency to backtrack.
• Figure 3: Learning methods for two kinds of relationships
• Figure 4: The distribution of trajectory lengths predicted on the val seen and val unseen splits of REVERIE dataset.
• Figure 5: Visualization of attention maps and navigation examples