CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation

TL;DR

CoNav tackles the fusion of 2D images, 3D point clouds, and natural language in embodied navigation by introducing Cross-Modal Belief Alignment, which shares textual spatial hypotheses from a frozen 3D-text LLM to guide a pretrained image–text navigation agent. The approach decouples modalities, uses a Communication Interface for model-level fusions, and applies lightweight fine-tuning on a compact 2D–3D–text corpus to resolve conflicts between modalities. Empirically, CoNav achieves state-of-the-art results on VLN benchmarks (R2R, CVDN, REVERIE, SOON) and spatial-reasoning tasks (ScanQA, SQA3D), while producing shorter, more efficient paths (higher SPL) in many cases. The work demonstrates a practical path toward leveraging limited triple-modality data to enhance embodied navigation through structured, textual guidance from a 3D understanding module.

Abstract

Embodied navigation demands comprehensive scene understanding and precise spatial reasoning. While image-text models excel at interpreting pixel-level color and lighting cues, 3D-text models capture volumetric structure and spatial relationships. However, unified fusion approaches that jointly fuse 2D images, 3D point clouds, and textual instructions face challenges in limited availability of triple-modality data and difficulty resolving conflicting beliefs among modalities. In this work, we introduce CoNav, a collaborative cross-modal reasoning framework where a pretrained 3D-text model explicitly guides an image-text navigation agent by providing structured spatial-semantic knowledge to resolve ambiguities during navigation. Specifically, we introduce Cross-Modal Belief Alignment, which operationalizes this cross-modal guidance by simply sharing textual hypotheses from the 3D-text model to the navigation agent. Through lightweight fine-tuning on a small 2D-3D-text corpus, the navigation agent learns to integrate visual cues with spatial-semantic knowledge derived from the 3D-text model, enabling effective reasoning in embodied navigation. CoNav achieves significant improvements on four standard embodied navigation benchmarks (R2R, CVDN, REVERIE, SOON) and two spatial reasoning benchmarks (ScanQA, SQA3D). Moreover, under close navigation Success Rate, CoNav often generates shorter paths compared to other methods (as measured by SPL), showcasing the potential and challenges of fusing data from different modalities in embodied navigation. Project Page: https://oceanhao.github.io/CoNav/

Figures (8)

• Figure 1: (a) Image–text navigation agents rely on only visual cues may fail when dealing with tasks involving spatial distance. Although 3D–text models excel in spatial reasoning, they are less sensitive to textures (e.g., map and painting on the wall). Embodied navigation often demands both visual cues and spatial geometry for reasoning. In the scene of Figure, the agent need to use visual cues to locate a world map and then apply spatial distance reasoning to determine which bedroom is nearest to that map. (b) Fusion of image and 3D can be achieved through three methods. Past unified fusion approaches often employed multimodal fusion at the feature level to complete cross-modal reasoning. By enabling text-based information sharing at the model level, CoNav achieves higher-level cross-modal reasoning.
• Figure 2: Our CoNav collaborative framework comprises an image–text navigation agent and a 3D–text model. The pretrained 3D–text model explicitly guides the image–text agent by providing structured spatial-semantic knowledge to resolve navigation ambiguities. The core of CoNav is the Cross-Modal Belief Alignment, which employs a Communication Interface to facilitate the straightforward sharing of textual hypotheses from our 3D–text model with our navigation agent. Through lightweight fine-tuning on a small 2D–3D–text corpus, CoNav learns to integrate visual cues with spatial-semantic knowledge derived from our 3D–text model.
• Figure 3: Visualization of CoNav in R2R. In Figure (a), the baseline image-text navigation agent rely on only visual cues, stops prematurely before reaching the target. This is likely caused by the image-text agent being unaware of if it is in the middle of the room, mistakenly stopping when observing the ping pong table. CoNav, on the other hand makes the correct decision. In Figure (b), there are two sinks. During navigation, the agent first encounters the incorrect sink and then see the second, correct sink. The image-text agent moves towards the first incorrect sink as soon as it observes it. This may be caused by the agent's insensitivity to directions (e.g., left/right) and its inability to determine when it has passed the left room. With Collaborative Cross-Modal Reasoning, CoNav finds the correct path. In Figure (c), the image-text navigation agent stops at the closer end of the bar without reaching the far end. This may be due to pixel-level color information not providing distance cues, whereas the 3D geometric structure can provide such distance guidance.
• Figure 4: (a) Structure of our auxiliar 3D-text model. (b) Progressive Curriculum Learning training paradigm of our 3D-text model.
• Figure 5: More Qualitative results of CoNav for different Embodied Navigation task.