Visual Representations for Semantic Target Driven Navigation

TL;DR

The paper tackles how visual representations influence target-driven semantic navigation. It proposes using high-level semantic features from segmentation and detection to train navigation policies, enabling training across real and synthetic data without domain adaptation. A POMDP-based framework with an LSTM-based policy and imitation learning supervises action choices via a cost that encodes progress toward the target. Empirical results on the Active Vision Dataset (augmented with SunCG data) show 54% success in unseen environments, outperforming a non-learning baseline by about 8%, and demonstrate the value of semantic representations for robust navigation.

Abstract

What is a good visual representation for autonomous agents? We address this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a complex environment to a target object, e.g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues. We propose to using high level semantic and contextual features including segmentation and detection masks obtained by off-the-shelf state-of-the-art vision as observations and use deep network to learn the navigation policy. This choice allows using additional data, from orthogonal sources, to better train different parts of the model the representation extraction is trained on large standard vision datasets while the navigation component leverages large synthetic environments for training. This combination of real and synthetic is possible because equitable feature representations are available in both (e.g., segmentation and detection masks), which alleviates the need for domain adaptation. Both the representation and the navigation policy can be readily applied to real non-synthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach gets successfully to the target in 54% of the cases in unexplored environments, compared to 46% for non-learning based approach, and 28% for the learning-based baseline.

Figures (8)

• Figure 1: Visualization of detections (left), segmentations (middle) and depth (right). While for real data (top) we use off-the-shelf detector and segmenter, the output on the simulated data (bottom) comes from the labels in this data and the renderer.
• Figure 2: Model Overview: RNN takes concatenation of joint representation of observations and target in addition to previous action and success indicator. The model predicts the cost $v(o,a; c)$ of taking action $a$ at the current state. Finally, the controller takes the action with the lowest cost (see. (\ref{['eq:controller']})).
• Figure 3: Success rates of the models on similar (blue) and different environment split (red) of AVD.
• Figure 4: Success rate of our model with different representations when trained using AVD only (blue) or AVD and SunCG (red).
• Figure 5: Success rates of a feedforward vs recurrent model using AVD only (denoted by real) or AVD and SunCG.