NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation

TL;DR

NavQ proposes a foresighted VLN agent by learning a Q-feature that summarizes long-horizon future observations for each candidate action. A self-supervised Q-model is trained on unlabeled trajectories, and a Future Encoder transforms these Q-features into future-aware action scores that are fused with history-based scores in an A*-style planning framework. The approach relies on three-stage training (Q-model pretraining, offline agent pretraining with progress signals, and online finetuning with a pseudo-expert) and leverages text-based Q-prediction to improve generalization. Experiments on REVERIE and SOON show consistent gains over strong baselines, with additional unlabeled data and GT Q-features providing further upper-bound improvements, illustrating practical gains in foresighted embodied navigation.

Abstract

In this work we concentrate on the task of goal-oriented Vision-and-Language Navigation (VLN). Existing methods often make decisions based on historical information, overlooking the future implications and long-term outcomes of the actions. In contrast, we aim to develop a foresighted agent. Specifically, we draw upon Q-learning to train a Q-model using large-scale unlabeled trajectory data, in order to learn the general knowledge regarding the layout and object relations within indoor scenes. This model can generate a Q-feature, analogous to the Q-value in traditional Q-network, for each candidate action, which describes the potential future information that may be observed after taking the specific action. Subsequently, a cross-modal future encoder integrates the task-agnostic Q-feature with navigation instructions to produce a set of action scores reflecting future prospects. These scores, when combined with the original scores based on history, facilitate an A*-style searching strategy to effectively explore the regions that are more likely to lead to the destination. Extensive experiments conducted on widely used goal-oriented VLN datasets validate the effectiveness of the proposed method.

Figures (7)

• Figure 1: Left: the motivation of the proposed method. We generate cumulative Q-feature for each candidate action, which represents the future outcomes of choosing the action and enables foresighted navigation decisions. Right: a high-level comparison among the decision making processes of (a) methods based on imagining neighborhood observations, (b) methods based on a world model, and (c) our proposed method. Our Q-model is capable of forecasting the long-horizon future without time-consuming rollouts.
• Figure 2: An overview of the proposed model. The gray modules are inherited from the baseline model DUET, while the orange ones are introduced by this work.
• Figure 3: The design of our Q-model. Given a randomly sampled partial trajectory, the Q-model takes the observation and action features along the way as input, and predicts the Q-features for the candidate actions (f, g, h, i) at current node. The ground-truth Q-feature is the cumulated feature of all possible future nodes. We present a visualization of the cumulated nodes for each candidate action. The intensity of the red color on the node reflects the magnitude of the decay factor ($\gamma^t$) for cumulation.
• Figure 4: A comparison among different training techniques for the Q-model. We plot the MSE loss on the val-unseen scenes during the training process.
• Figure 5: A qualitative comparison of our method and the baseline agent. In both examples, the NavQ agent performs the instruction correctly while the baseline agent fails.