Context-Aware Replanning with Pre-explored Semantic Map for Object Navigation

TL;DR

Context-Aware Replanning (CARe) is introduced, which estimates map uncertainty through confidence scores and multi-view consistency, enabling the agent to revise erroneous decisions stemming from inaccurate maps without requiring additional labels.

Abstract

Pre-explored Semantic Maps, constructed through prior exploration using visual language models (VLMs), have proven effective as foundational elements for training-free robotic applications. However, existing approaches assume the map's accuracy and do not provide effective mechanisms for revising decisions based on incorrect maps. To address this, we introduce Context-Aware Replanning (CARe), which estimates map uncertainty through confidence scores and multi-view consistency, enabling the agent to revise erroneous decisions stemming from inaccurate maps without requiring additional labels. We demonstrate the effectiveness of our proposed method by integrating it with two modern mapping backbones, VLMaps and OpenMask3D, and observe significant performance improvements in object navigation tasks. More details can be found on the project page: https://care-maps.github.io/

Figures (6)

• Figure 1: Motivation. When the initial map query fails, high-confidence regions also tend to fail due to biases in visual perception. (Query: Table, Grounding: Chair)
• Figure 2: Method Overview.
• Figure 3: Entropy vs. K. If a model is biased, selecting low-entropy candidates from it might further reinforce the bias and ultimately degrade performance. This figure illustrates the performance of choosing minimum entropy targets from top-k confidence candidates.
• Figure 4: KL Divergence vs. K. When the multi-view prediction is noisy, it's more likely to provide false information. This figure depicts the performance of selecting the least consistent candidate (i.e., maximum mean pairwise KL divergence) from top-k confidence candidates.
• Figure 5: Latency for Top-k Confidence.