Prioritized Semantic Learning for Zero-shot Instance Navigation

TL;DR

This work tackles zero-shot instance navigation by extending zero-shot object navigation to instances described by rich text or images. It introduces Prioritized Semantic Learning (PSL), a lightweight, CLIP-backed framework with a semantic perception module, entropy-based goal-view selection, and perspective reward relaxation to emphasize semantic alignment over exact pixel matching. A semantic expansion inference scheme preserves semantic granularity between training and testing, and an InstanceNav task in HM3D is introduced to evaluate open-vocabulary, instance-level goals. PSL achieves significant gains over prior methods on both ObjectNav and InstanceNav, including a substantial SR improvement on ObjectNav, and demonstrates strong performance without heavy LLM-based reasoning, highlighting practical efficiency and scalable semantic understanding for zero-shot navigation.

Abstract

We study zero-shot instance navigation, in which the agent navigates to a specific object without using object annotations for training. Previous object navigation approaches apply the image-goal navigation (ImageNav) task (go to the location of an image) for pretraining, and transfer the agent to achieve object goals using a vision-language model. However, these approaches lead to issues of semantic neglect, where the model fails to learn meaningful semantic alignments. In this paper, we propose a Prioritized Semantic Learning (PSL) method to improve the semantic understanding ability of navigation agents. Specifically, a semantic-enhanced PSL agent is proposed and a prioritized semantic training strategy is introduced to select goal images that exhibit clear semantic supervision and relax the reward function from strict exact view matching. At inference time, a semantic expansion inference scheme is designed to preserve the same granularity level of the goal semantic as training. Furthermore, for the popular HM3D environment, we present an Instance Navigation (InstanceNav) task that requires going to a specific object instance with detailed descriptions, as opposed to the Object Navigation (ObjectNav) task where the goal is defined merely by the object category. Our PSL agent outperforms the previous state-of-the-art by 66% on zero-shot ObjectNav in terms of success rate and is also superior on the new InstanceNav task. Code will be released at https://github.com/XinyuSun/PSL-InstanceNav.

Figures (4)

• Figure 1: Pilot studies on four navigation agents: Semantic-Non-dominant (SN) agent, Semantic-Dominant (SD) agent, ZSON zson agent, and our PSL agent. (a) The agent trained on $\texttt{ImageNav}$ task does not necessarily need to learn the semantic information to obtain a high success rate. (b)-(c) Our PSL agent achieves both strong semantic perception and navigation capacity.
• Figure 2: Overview of our PSL. 1) During $\texttt{ImageNav}$ pre-training, we provide clear semantic supervision signals with the Prioritized Semantic Training strategy. 2) Our PSL agent exploits the Semantic Perception Module (SPM) to achieve both strong semantic understanding and navigation capacity. 3) During inference, a Semantic Expansion Inference scheme is incorporated to ensure the same semantic granularity of the goal-embedding between training and testing.
• Figure 3: Illustration of the perspective relaxation and goal view selection approach in our training strategy. The goal view image set is expanded with additional yaw and tilt views using perspective relaxation. Top-10 views with minimum entropy are circled in red with their ranking.
• Figure 4: Comparison of the $\texttt{InstanceNav}$ task with the $\texttt{ObjectNav}$ task. The $\texttt{ObjectNav}$ task allows the agent to navigate to any chair in the scene, while the $\texttt{InstanceNav}$ task has only one target object specified by detailed attribute descriptions.