Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProFocus: Proactive Perception and Focused Reasoning in Vision-and-Language Navigation

Wei Xue, Mingcheng Li, Xuecheng Wu, Jingqun Tang, Dingkang Yang, Lihua Zhang

TL;DR

ProFocus is a training-free progressive framework that unifies Proactive Perception and Focus Reasoning through collaboration between large language models (LLMs) and vision-language models (VLMs) and vision-language models (VLMs).

Abstract

Vision-and-Language Navigation (VLN) requires agents to accurately perceive complex visual environments and reason over navigation instructions and histories. However, existing methods passively process redundant visual inputs and treat all historical contexts indiscriminately, resulting in inefficient perception and unfocused reasoning. To address these challenges, we propose \textbf{ProFocus}, a training-free progressive framework that unifies \underline{Pro}active Perception and \underline{Focus}ed Reasoning through collaboration between large language models (LLMs) and vision-language models (VLMs). For proactive perception, ProFocus transforms panoramic observations into structured ego-centric semantic maps, enabling the orchestration agent to identify missing visual information needed for reliable decision-making, and to generate targeted visual queries with corresponding focus regions that guide the perception agent to acquire the required observations. For focused reasoning, we propose Branch-Diverse Monte Carlo Tree Search (BD-MCTS) to identify top-$k$ high-value waypoints from extensive historical candidates. The decision agent focuses reasoning on the historical contexts associated with these waypoints, rather than considering all historical waypoints equally. Extensive experiments validate the effectiveness of ProFocus, achieving state-of-the-art performance among zero-shot methods on R2R and REVERIE benchmarks.

ProFocus: Proactive Perception and Focused Reasoning in Vision-and-Language Navigation

TL;DR

ProFocus is a training-free progressive framework that unifies Proactive Perception and Focus Reasoning through collaboration between large language models (LLMs) and vision-language models (VLMs) and vision-language models (VLMs).

Abstract

Vision-and-Language Navigation (VLN) requires agents to accurately perceive complex visual environments and reason over navigation instructions and histories. However, existing methods passively process redundant visual inputs and treat all historical contexts indiscriminately, resulting in inefficient perception and unfocused reasoning. To address these challenges, we propose \textbf{ProFocus}, a training-free progressive framework that unifies \underline{Pro}active Perception and \underline{Focus}ed Reasoning through collaboration between large language models (LLMs) and vision-language models (VLMs). For proactive perception, ProFocus transforms panoramic observations into structured ego-centric semantic maps, enabling the orchestration agent to identify missing visual information needed for reliable decision-making, and to generate targeted visual queries with corresponding focus regions that guide the perception agent to acquire the required observations. For focused reasoning, we propose Branch-Diverse Monte Carlo Tree Search (BD-MCTS) to identify top- high-value waypoints from extensive historical candidates. The decision agent focuses reasoning on the historical contexts associated with these waypoints, rather than considering all historical waypoints equally. Extensive experiments validate the effectiveness of ProFocus, achieving state-of-the-art performance among zero-shot methods on R2R and REVERIE benchmarks.
Paper Structure (14 sections, 13 equations, 3 figures, 4 tables)

This paper contains 14 sections, 13 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-and-Language Navigation
  4. Perception and Contextual Reasoning
  5. Methodology
  6. Overview
  7. Reasoning-Guided Proactive Perception
  8. Focused Reasoning via Branch-Diverse MCTS
  9. Experiments
  10. Experimental Setup
  11. Performance Analysis and Comparisons
  12. Ablation Studies
  13. Qualitative Analysis
  14. Conclusion

Figures (3)

  • Figure 1: Overview of the ProFocus framework. ProFocus consists of two core mechanisms: Proactive Perception, which selectively acquires instruction-relevant visual evidence through a closed perception--reasoning loop, and Focused Reasoning, which leverages BD-MCTS to highlight top-$k$ high-value waypoints for effective focused reasoning.
  • Figure 2: Performance comparison on the 30 longest navigation episodes from R2R validation unseen set.
  • Figure 3: Qualitative case study demonstrating proactive perception and focused reasoning.Left: Proactive perception enables accurate semantic value estimation by acquiring fine-grained visual details. Right: BD-MCTS-guided top-$k$ focused reasoning enables error correction through global historical awareness.