Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion

Xinyu Gao, Gang Chen, Javier Alonso-Mora

TL;DR

BEACON predicts an ego-centric Bird's-Eye View (BEV) affordance heatmap over a bounded local region including occluded areas, which improves the accuracy averaged across geodesic thresholds by 22.74 percentage points over the state-of-the-art image-space baseline on the validation subset with occluded target locations.

Abstract

Language-conditioned local navigation requires a robot to infer a nearby traversable target location from its current observation and an open-vocabulary, relational instruction. Existing vision-language spatial grounding methods usually rely on vision-language models (VLMs) to reason in image space, producing 2D predictions tied to visible pixels. As a result, they struggle to infer target locations in occluded regions, typically caused by furniture or moving humans. To address this issue, we propose BEACON, which predicts an ego-centric Bird's-Eye View (BEV) affordance heatmap over a bounded local region including occluded areas. Given an instruction and surround-view RGB-D observations from four directions around the robot, BEACON predicts the BEV heatmap by injecting spatial cues into a VLM and fusing the VLM's output with depth-derived BEV features. Using an occlusion-aware dataset built in the Habitat simulator, we conduct detailed experimental analysis to validate both our BEV space formulation and the design choices of each module. Our method improves the accuracy averaged across geodesic thresholds by 22.74 percentage points over the state-of-the-art image-space baseline on the validation subset with occluded target locations. Our project page is: https://xin-yu-gao.github.io/beacon.

BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion

TL;DR

BEACON predicts an ego-centric Bird's-Eye View (BEV) affordance heatmap over a bounded local region including occluded areas, which improves the accuracy averaged across geodesic thresholds by 22.74 percentage points over the state-of-the-art image-space baseline on the validation subset with occluded target locations.

Abstract

Language-conditioned local navigation requires a robot to infer a nearby traversable target location from its current observation and an open-vocabulary, relational instruction. Existing vision-language spatial grounding methods usually rely on vision-language models (VLMs) to reason in image space, producing 2D predictions tied to visible pixels. As a result, they struggle to infer target locations in occluded regions, typically caused by furniture or moving humans. To address this issue, we propose BEACON, which predicts an ego-centric Bird's-Eye View (BEV) affordance heatmap over a bounded local region including occluded areas. Given an instruction and surround-view RGB-D observations from four directions around the robot, BEACON predicts the BEV heatmap by injecting spatial cues into a VLM and fusing the VLM's output with depth-derived BEV features. Using an occlusion-aware dataset built in the Habitat simulator, we conduct detailed experimental analysis to validate both our BEV space formulation and the design choices of each module. Our method improves the accuracy averaged across geodesic thresholds by 22.74 percentage points over the state-of-the-art image-space baseline on the validation subset with occluded target locations. Our project page is: https://xin-yu-gao.github.io/beacon.
Paper Structure (23 sections, 4 equations, 4 figures, 3 tables)

This paper contains 23 sections, 4 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Spatial Grounding in Robotics
  4. VLMs with Local 3D or Ego-Centric Multi-View Inputs
  5. BEV Representations and VLM Alignment
  6. Problem Formulation
  7. Methodology
  8. Ego-Aligned Vision-Language Model
  9. Ego-Centric 3D Position Encoding
  10. Navigation Task Token Interface
  11. Stage-1 Auto-Derived Ego-Centric Instruction Tuning
  12. Geometry-Aware Bird's-Eye View Encoder
  13. Post-Fusion Affordance Decoder
  14. Geodesic Target Region Supervision
  15. Experimental Setup
  16. ...and 8 more sections

Figures (4)

  • Figure 1: BEACON predicts an ego-centric Bird's-Eye View (BEV) affordance heatmap for language-conditioned local navigation, which is better suited to occluded targets than the state-of-the-art image-space grounding method.
  • Figure 2: Examples of language-conditioned local navigation under occlusion. The blue boxes mark the robot, the red boxes highlight humans and objects that cause occlusions, and the green boxes indicate target regions.
  • Figure 3: BEACON overview. Stage 1 performs auto-derived ego-centric instruction tuning with ego-centric 3D position encoding to train the Ego-Aligned VLM. Stage 2 initializes the Ego-Aligned VLM weights from Stage 1, combines the resulting instruction-conditioned output with Geometry-Aware BEV features, and predicts an ego-centric BEV navigation affordance heatmap via a Post-Fusion Affordance Decoder. The two stages use different supervision signals, and inference selects the navigation target by taking the argmax.
  • Figure 4: Qualitative examples of language-conditioned navigation affordance prediction, comparing BEACON’s BEV affordance predictions with the image-space baselines RoboPoint yuan2025robopoint and RoboRefer zhouroborefer. Target regions are shown in green.