Transformers for Image-Goal Navigation

TL;DR

This paper addresses image-goal navigation by applying a decoder-only Transformer trained with goal-conditioned behavior cloning to predict long-horizon actions from sequences of observations and goal images. The model jointly encodes the goal, observations, and past actions and autoregressively generates actions, avoiding costly online interaction with environments. Experiments in Gibson/Habitat demonstrate that GCBC with a 27M-parameter Transformer outperforms other BC baselines while acknowledging a gap to online RL methods, and highlight practical insights like the benefits of top-$k$ action sampling and limitations due to occlusions and dataset biases. The work suggests future improvements through perceptual pretraining, longer context, and sim-to-real fine-tuning to enhance robustness and scalability of image-goal navigation systems.

Abstract

Visual perception and navigation have emerged as major focus areas in the field of embodied artificial intelligence. We consider the task of image-goal navigation, where an agent is tasked to navigate to a goal specified by an image, relying only on images from an onboard camera. This task is particularly challenging since it demands robust scene understanding, goal-oriented planning and long-horizon navigation. Most existing approaches typically learn navigation policies reliant on recurrent neural networks trained via online reinforcement learning. However, training such policies requires substantial computational resources and time, and performance of these models is not reliable on long-horizon navigation. In this work, we present a generative Transformer based model that jointly models image goals, camera observations and the robot's past actions to predict future actions. We use state-of-the-art perception models and navigation policies to learn robust goal conditioned policies without the need for real-time interaction with the environment. Our model demonstrates capability in capturing and associating visual information across long time horizons, helping in effective navigation. NOTE: This work was submitted as part of a Master's Capstone Project and must be treated as such. This is still an early work in progress and not the final version.

Figures (10)

• Figure 1: The Image-Goal navigation task. The robot receives a goal image (shown in the green dotted box), and must navigate to find the goal image using only visual observations. In the figure above, the red camera marks the initial viewpoint, with "S" marking the robot's initial position. The robot navigates to the point marked "G" described by the goal image, i.e., the view from the green camera. The dashed lines out of the camera indicate the camera's approximate field of view. The yellow arrows indicate one possible path from "S" to "G". The figure above shows the "Pablo" scene from the Gibson training dataset, obtained from armeni20193d.
• Figure 2: The full architecture of our model. The goal image $I_G$ and observation images $I_1, I_2,...I_T$ are embedded into tokens $E_{I_G}$ and $E_{I_1}, E_{I_2},....E_{I_T}$ respectively using a pre-trained DINOv2 model, actions $a_1, a_2,...a_T$ are passed to a specific lookup embedding. To enable autoregressive generation, the input actions are shifted right by one step. Image and action embeddings are interleaved to form input embeddings $E_1, E_2,...E_T$. Learnable position embeddings $p_1, p_2,...p_T$are added before passing all inputs to a Transformer decoder with a causal attention mask. The decoder outputs logits, corresponding to the most likely action. An actor head samples the actions from these logits.
• Figure 3: Model accuracy at different start indices of non-overlapping windows of input trajectory, for curved (\ref{['fig:window_accuracy_curved']}) and straight episodes (\ref{['fig:window_accuracy_straight']}). Results indicate model accuracy is higher at the episode's beginning and lower near its middle.
• Figure 4: Distribution of episode lengths across various difficulty splits from the "Gibson-curved" training set."Easy" episodes are considerably shorter than "Medium" and "Hard" episodes.
• Figure 5: Model's performance on a successful "easy" difficulty episode from the "curved" split of the Gibson test set.