Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Language Models

TL;DR

Co-NavGPT tackles multi-robot visual target navigation by placing a Vision-Language Model as a global planner that reasons over a fused semantic map and assigns frontiers to multiple robots. The system constructs per-robot 3D maps and a 2D exploration grid, then uses multi-modal prompts to solicit semantically informed frontier allocations, with a Fast Marching Method-based local policy executing safe, short-horizon paths. Key contributions include a VLM-driven frontier assignment mechanism, a modular map-then-prompt planning pipeline, and comprehensive validation in HM3D and real-world robot experiments showing improved SR and SPL without task-specific training. The work highlights the practical potential of VLMs to coordinate complex multi-agent exploration and search tasks in unknown environments, paving the way for tighter integration and real-time interactive decision-making in embodied systems.

Abstract

Visual target navigation is a critical capability for autonomous robots operating in unknown environments, particularly in human-robot interaction scenarios. While classical and learning-based methods have shown promise, most existing approaches lack common-sense reasoning and are typically designed for single-robot settings, leading to reduced efficiency and robustness in complex environments. To address these limitations, we introduce Co-NavGPT, a novel framework that integrates a Vision Language Model (VLM) as a global planner to enable common-sense multi-robot visual target navigation. Co-NavGPT aggregates sub-maps from multiple robots with diverse viewpoints into a unified global map, encoding robot states and frontier regions. The VLM uses this information to assign frontiers across the robots, facilitating coordinated and efficient exploration. Experiments on the Habitat-Matterport 3D (HM3D) demonstrate that Co-NavGPT outperforms existing baselines in terms of success rate and navigation efficiency, without requiring task-specific training. Ablation studies further confirm the importance of semantic priors from the VLM. We also validate the framework in real-world scenarios using quadrupedal robots. Supplementary video and code are available at: https://sites.google.com/view/co-navgpt2.

Figures (4)

• Figure 1: Two robots visual target navigation example. When multiple unexplored frontiers are detected, the vision language model assigns the frontier for each robot based on the current observation and the target object.
• Figure 2: System architecture of the proposed multi-robot navigation framework. Each robot processes RGB-D observations to generate a local point cloud map, which is then merged into a global 3D map. The merged map, robot states, and candidate frontiers are encoded into a prompt and passed to a vision-language model, which acts as a global planner to assign frontier goals to each robot. A local policy then computes paths to the assigned frontiers based on the obstacle map, enabling coordinated exploration and target search.
• Figure 3: Visualization of the visual target navigation process in the Habitat simulator using two robots to locate a bed. The top row shows first-person RGB observations from both robots (Agent 0 in red, Agent 1 in blue) at different time steps. The bottom row presents the corresponding 2D exploration map and 3D point cloud map. The red and blue lines represent the respective paths of Agent 0 and Agent 1. Red and blue dots denote frontier goals assigned by the vision-language model.
• Figure 4: Real-world multi-robot visual target navigation. Each scene shows the first-person RGB images, the exploration map, and the 3D point cloud map. In (a), the robots search for a sink; (b) search for a person; (c) search for a chair. Red and blue lines denote the trajectories of the two robots.