Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Network-Constrained Policy Optimization for Adaptive Multi-agent Vehicle Routing

Fazel Arasteh, Arian Haghparast, Manos Papagelis

TL;DR

This paper tackles dynamic vehicle routing in urban road networks by formulating a network-aware multi-agent reinforcement learning approach. It introduces Adaptive Navigation (AN), a fully decentralized MARL system with Graph Attention Networks to enable intersection-level coordination, and Hierarchical Hub-based Adaptive Navigation (HHAN), a scalable extension using centralized training with Attentive QMIX for hub-level coordination and SPF for micro-routing inside hub regions. A novel Z-order destination encoding preserves spatial locality while remaining compact, and the A-QMIX framework handles asynchronous decisions through attention-based aggregation, enabling scalable coordination across large networks. Empirical results on synthetic grids and real maps (Toronto, Manhattan) show AN reduces average travel time and achieves 100% routing success, while HHAN scales to hundreds of intersections and yields substantial improvements under heavy traffic, demonstrating practical potential for scalable, congestion-aware routing in intelligent transportation systems. The work provides open-source code and introduces a principled combination of network-aware MARL, graph-based coordination, and hierarchical abstractions that can inform future deployments in urban mobility and related networked systems.

Abstract

Traffic congestion in urban road networks leads to longer trip times and higher emissions, especially during peak periods. While the Shortest Path First (SPF) algorithm is optimal for a single vehicle in a static network, it performs poorly in dynamic, multi-vehicle settings, often worsening congestion by routing all vehicles along identical paths. We address dynamic vehicle routing through a multi-agent reinforcement learning (MARL) framework for coordinated, network-aware fleet navigation. We first propose Adaptive Navigation (AN), a decentralized MARL model where each intersection agent provides routing guidance based on (i) local traffic and (ii) neighborhood state modeled using Graph Attention Networks (GAT). To improve scalability in large networks, we further propose Hierarchical Hub-based Adaptive Navigation (HHAN), an extension of AN that assigns agents only to key intersections (hubs). Vehicles are routed hub-to-hub under agent control, while SPF handles micro-routing within each hub region. For hub coordination, HHAN adopts centralized training with decentralized execution (CTDE) under the Attentive Q-Mixing (A-QMIX) framework, which aggregates asynchronous vehicle decisions via attention. Hub agents use flow-aware state features that combine local congestion and predictive dynamics for proactive routing. Experiments on synthetic grids and real urban maps (Toronto, Manhattan) show that AN reduces average travel time versus SPF and learning baselines, maintaining 100% routing success. HHAN scales to networks with hundreds of intersections, achieving up to 15.9% improvement under heavy traffic. These findings highlight the potential of network-constrained MARL for scalable, coordinated, and congestion-aware routing in intelligent transportation systems.

Network-Constrained Policy Optimization for Adaptive Multi-agent Vehicle Routing

TL;DR

This paper tackles dynamic vehicle routing in urban road networks by formulating a network-aware multi-agent reinforcement learning approach. It introduces Adaptive Navigation (AN), a fully decentralized MARL system with Graph Attention Networks to enable intersection-level coordination, and Hierarchical Hub-based Adaptive Navigation (HHAN), a scalable extension using centralized training with Attentive QMIX for hub-level coordination and SPF for micro-routing inside hub regions. A novel Z-order destination encoding preserves spatial locality while remaining compact, and the A-QMIX framework handles asynchronous decisions through attention-based aggregation, enabling scalable coordination across large networks. Empirical results on synthetic grids and real maps (Toronto, Manhattan) show AN reduces average travel time and achieves 100% routing success, while HHAN scales to hundreds of intersections and yields substantial improvements under heavy traffic, demonstrating practical potential for scalable, congestion-aware routing in intelligent transportation systems. The work provides open-source code and introduces a principled combination of network-aware MARL, graph-based coordination, and hierarchical abstractions that can inform future deployments in urban mobility and related networked systems.

Abstract

Traffic congestion in urban road networks leads to longer trip times and higher emissions, especially during peak periods. While the Shortest Path First (SPF) algorithm is optimal for a single vehicle in a static network, it performs poorly in dynamic, multi-vehicle settings, often worsening congestion by routing all vehicles along identical paths. We address dynamic vehicle routing through a multi-agent reinforcement learning (MARL) framework for coordinated, network-aware fleet navigation. We first propose Adaptive Navigation (AN), a decentralized MARL model where each intersection agent provides routing guidance based on (i) local traffic and (ii) neighborhood state modeled using Graph Attention Networks (GAT). To improve scalability in large networks, we further propose Hierarchical Hub-based Adaptive Navigation (HHAN), an extension of AN that assigns agents only to key intersections (hubs). Vehicles are routed hub-to-hub under agent control, while SPF handles micro-routing within each hub region. For hub coordination, HHAN adopts centralized training with decentralized execution (CTDE) under the Attentive Q-Mixing (A-QMIX) framework, which aggregates asynchronous vehicle decisions via attention. Hub agents use flow-aware state features that combine local congestion and predictive dynamics for proactive routing. Experiments on synthetic grids and real urban maps (Toronto, Manhattan) show that AN reduces average travel time versus SPF and learning baselines, maintaining 100% routing success. HHAN scales to networks with hundreds of intersections, achieving up to 15.9% improvement under heavy traffic. These findings highlight the potential of network-constrained MARL for scalable, coordinated, and congestion-aware routing in intelligent transportation systems.

Paper Structure

This paper contains 47 sections, 35 equations, 12 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Current approaches and limitations
  4. Problem formulation
  5. Our approach
  6. Contributions
  7. Organization
  8. Problem Definition
  9. Network Model
  10. Adaptive Navigation Problem
  11. Methodology
  12. Adaptive Navigation (AN)
  13. MARL Formulation
  14. Destination Representation
  15. Algorithm Sketch
  16. ...and 32 more sections

Figures (12)

  • Figure 1: Hard to separate destinations.
  • Figure 2: Collaborative policies.
  • Figure 3: Z-order curve (Morton Space Filtering).
  • Figure 4: Adaptive Navigation (AN) model architecture showing the decentralized MARL approach where each intersection has an agent that processes routing queries using destination embeddings and GAT-aggregated network states. The GAT layers enable neighborhood information sharing for implicit coordination between agents, while each agent makes independent routing decisions based on local Q-networks trained with intertwined Q-learning updates.
  • Figure 5: Hierarchical Hub-based Adaptive Navigation (HHAN) model architecture implementing centralized training with decentralized execution (CTDE) using the A-QMIX framework. Hub agents process local observations including destination embeddings and flow-aware features, making asynchronous routing decisions that are aggregated through an attention mechanism. The mixing network ensures monotonic value function combination while enabling coordinated learning across the hub-based network structure.
  • ...and 7 more figures

Theorems & Definitions (14)

  • Definition 1: $q$: routing query
  • Definition 2: $NH(r)$: next-hop road set
  • Definition 3: $resp(q)$: routing response
  • Definition 4: $trip$: trip of vehicle $vc$
  • Definition 5: $path(trip)$: path of a trip
  • Definition 6: $tt(p)$: travel time of path $p$
  • Definition 7: $RS$: Routing Success
  • Definition 8: $AVTT$: average travel time
  • Definition 9: Locality of access
  • Definition 10: Adaptive Navigation Problem
  • ...and 4 more