ORION: Option-Regularized Deep Reinforcement Learning for Cooperative Multi-Agent Online Navigation

TL;DR

ORION addresses cooperative multi-agent online navigation under partial observability by fusing prior and online maps through a graph encoder and driving decisions with an option-regularized policy and critic. Its dual-stage cooperation enables agents to help teammates after reaching targets, improving the team makespan while maintaining real-time performance. Key contributions include the informative graph observations, an option-critic framework with termination and FSM-based transitions, and a privileged-information critic for stable long-horizon value estimation. The approach demonstrates strong gains in simulated, Gazebo, and real-world experiments, highlighting practical applicability for scalable, decentralized cooperative navigation.

Abstract

Existing methods for multi-agent navigation typically assume fully known environments, offering limited support for partially known scenarios such as warehouses or factory floors. There, agents may need to plan trajectories that balance their own path optimality with their ability to collect and share information about the environment that can help their teammates reach their own goals. To these ends, we propose ORION, a novel deep reinforcement learning framework for cooperative multi-agent online navigation in partially known environments. Starting from an imperfect prior map, ORION trains agents to make decentralized decisions, coordinate to reach their individual targets, and actively reduce map uncertainty by sharing online observations in a closed perception-action loop. We first design a shared graph encoder that fuses prior map with online perception into a unified representation, providing robust state embeddings under dynamic map discrepancies. At the core of ORION is an option-critic framework that learns to reason about a set of high-level cooperative modes that translate into sequences of low-level actions, allowing agents to switch between individual navigation and team-level exploration adaptively. We further introduce a dual-stage cooperation strategy that enables agents to assist teammates under map uncertainty, thereby reducing the overall makespan. Across extensive maze-like maps and large-scale warehouse environments, our simulation results show that ORION achieves high-quality, real-time decentralized cooperation over varying team sizes, outperforming state-of-the-art classical and learning-based baselines. Finally, we validate ORION on physical robot teams, demonstrating its robustness and practicality for real-world cooperative navigation.

Figures (6)

• Figure 1: Overview of ORION for cooperative multi-agent online navigation. Each agent is assigned a target and navigates over a prior map that may differ from the ground truth. Agents maintain/share (a) prior map, (b) current map, and (c) combined map that fuse prior/online sources to reason about partially changed environments. During navigation, agents not only pursue their own targets but also cooperate by sharing information and assisting others. For example, the red agent reaches its target early and then helps the yellow agent by exploring uncertain regions before returning. ORION enables such adaptive cooperation both before and after arrival on-goal, ultimately reducing the team's makespan by coordinating agents to contribute where they are most needed in a decentralized way.
• Figure 2: Option-regularized policy and multi-agent critic networks. The combined and current encoders fuse prior information with online observations into joint features. A termination head and option decoder then decide whether to maintain the current option or switch to a new valid one, while the waypoint decoder integrates the option feature with the current node feature to select a waypoint from the agent's neighboring nodes. In parallel, the critic network, conditioned on the states, actions, and selected options of all agents, provides centralized value estimates. During training, it leverages the ground-truth map as privileged information to estimate option--state values that capture the long-term team return.
• Figure 3: Comparison of travel distances on simulated maps. For each planner, each bar encodes three statistics: the top, middle, and bottom markers correspond to the makespan, the average travel distance, and the minimum distance within the team, respectively. Results are shown for teams of $3$, $4$, $5$, and $10$ agents.
• Figure 4: ROS Experiments. ORION yields more efficient decentralized coordination, as seen from each agent's local maps and the shared map, compared to ORION w/o option.
• Figure 5: Runtime performance in Gazebo simulation. ORION maintains real-time performance on graph updates (green) and network inference (blue) throughout execution, while the prior utility curve (orange) reflects how ORION incrementally verifies and corrects uncertain regions in the prior map during online navigation.