PIGEON: VLM-Driven Object Navigation via Points of Interest Selection

TL;DR

PIGEON tackles object navigation in unseen environments by balancing decision frequency with semantic reasoning. It introduces Points of Interest as snapshot memories and uses a VLM (PIGEON-VL) to select PoIs during exploration, while a lightweight low-level planner handles inter-PoI travel; RLVR facilitates training without manual Chain-of-Thought data. The approach yields state-of-the-art zero-shot performance on HM3D datasets and further gains when reinforced with RLVR, demonstrating robust, semantically guided navigation with reduced VLM calls. This combination enables efficient, interpretable navigation suitable for real-time deployment on embodied agents.

Abstract

Navigating to a specified object in an unknown environment is a fundamental yet challenging capability of embodied intelligence. However, current methods struggle to balance decision frequency with intelligence, resulting in decisions lacking foresight or discontinuous actions. In this work, we propose PIGEON: Point of Interest Guided Exploration for Object Navigation with VLM, maintaining a lightweight and semantically aligned snapshot memory during exploration as semantic input for the exploration strategy. We use a large Visual-Language Model (VLM), named PIGEON-VL, to select Points of Interest (PoI) formed during exploration and then employ a lower-level planner for action output, increasing the decision frequency. Additionally, this PoI-based decision-making enables the generation of Reinforcement Learning with Verifiable Reward (RLVR) data suitable for simulators. Experiments on classic object navigation benchmarks demonstrate that our zero-shot transfer method achieves state-of-the-art performance, while RLVR further enhances the model's semantic guidance capabilities, enabling deep reasoning during real-time navigation.

Figures (3)

• Figure 1: We introduce Points of Interest (PoI) during navigation process and utilize them directly as memories. The agent uses a low-level planner to navigate between PoIs, and decides the next best one to proceed to.
• Figure 2: The overview of our PIGEON. Green line: Navigation process along sequential waypoints. Blue line: During navigating between waypoints, PIGEON detects candidate objects to maintain directional PoIs and incrementally reveal unexplored regions to maintain non-directional PoIs. If the detection confidence with the goal object label surpasses a threshold $\tau_{sus}$, PIGEON invokes VLM with multi-view RGB snapshots of the object to determine whether the object corresponds to the goal object. Once arriving at a PoI, PIGEON archives outdated PoIs as historical records and queries VLM to identify the most promising PoI for exploration, where unselectable PoIs will be used as additional observations for VLM input. PIGEON then navigates to the selected PoI with low-level control algorithm. Red line: The performance of VLM can be enhanced with GRPO. After the model rolls out a group of text, PIGEON extract the selected PoI number in each output. The PoI with the shorter path to ground truth goal objects will obtain a higher reward through groupwise 0-1 normalization.
• Figure 3: Simulation result in HM3D dataset. At step 1, PIGEON spins around and determines mark 2 to be the most promising PoI. At step 2, PIGEON encounters an artwork of flower. The low level detector identifies the artwork as a potted plant, yet VLM denies it by exploiting information from environment. At step 3, PIGEON determines mark 2 to be the most promising PoI. At step 4, PIGEON confirms that it is facing a real potted plant, and calls stop.