BMG-Q: Localized Bipartite Match Graph Attention Q-Learning for Ride-Pooling Order Dispatch

TL;DR

The paper tackles large-scale ride-pooling order dispatch with interdependent vehicles by formulating a localized bipartite match interdependent MDP and introducing GATDDQN as the MARL backbone. It fuses a centralized dynamic ILP matcher with a posterior score function to balance exploration and mitigate overestimation, while employing graph sampling and gradient clipping to ensure scalability and stability. Empirical results on NYC taxi data show about a 10% gain in accumulative rewards and a >50% reduction in overestimation bias, along with robustness to fleet size and task variations. The work advances graph-based MARL for large transportation systems and offers a scalable framework for real-time, high-dimensional decision making in ride-pooling dispatch.

Abstract

This paper introduces Localized Bipartite Match Graph Attention Q-Learning (BMG-Q), a novel Multi-Agent Reinforcement Learning (MARL) algorithm framework tailored for ride-pooling order dispatch. BMG-Q advances ride-pooling decision-making process with the localized bipartite match graph underlying the Markov Decision Process, enabling the development of novel Graph Attention Double Deep Q Network (GATDDQN) as the MARL backbone to capture the dynamic interactions among ride-pooling vehicles in fleet. Our approach enriches the state information for each agent with GATDDQN by leveraging a localized bipartite interdependence graph and enables a centralized global coordinator to optimize order matching and agent behavior using Integer Linear Programming (ILP). Enhanced by gradient clipping and localized graph sampling, our GATDDQN improves scalability and robustness. Furthermore, the inclusion of a posterior score function in the ILP captures the online exploration-exploitation trade-off and reduces the potential overestimation bias of agents, thereby elevating the quality of the derived solutions. Through extensive experiments and validation, BMG-Q has demonstrated superior performance in both training and operations for thousands of vehicle agents, outperforming benchmark reinforcement learning frameworks by around 10% in accumulative rewards and showing a significant reduction in overestimation bias by over 50%. Additionally, it maintains robustness amidst task variations and fleet size changes, establishing BMG-Q as an effective, scalable, and robust framework for advancing ride-pooling order dispatch operations.

Figures (14)

• Figure 1: An illustrative example of decision-making process for two agents. At time $t$, two new requests are observed, including (a) Rider 1 from zone 3 to zone 5; and (b) Rider 2 from zone 9 to zone b. The platform then collaborative dispatches two vehicles: Vehicle 1 is assigned with action $a_{1,t}=1$ to integrate Rider 1 into its current route; while Vehicle 2 is idle and dispatched with action $a_{2,t} = 2$ to pickup Rider 2.
• Figure 2: Illustrative examples for different graph aggregation strategies. An empty ride-pooling agent in black is surrounded by eight vehicles in dark blue, each carrying passengers destined for distinct directions as indicated by the blue arrow. A new request arrives and intends to head southeast, as indicated by the black arrow. While GCN aggregators hamilton2017inductive in (a) or (b) can introduce significant inaccuracies, attention-based aggregation like GAT in (c) allows the empty agent to assign highest weights to its east and south neighbors, which are more compatible with the destination of the new order request.
• Figure 3: Visualization of GATDDQN algorithm pipeline. Step 1 constructs the localized bipartite match graph, passing the state features and adjacency matrix to Step 2. Step 2 performs GATDDQN and decision evaluation based on the inputs from Step 1.
• Figure 4: Dynamic ILP with posterior score function.
• Figure 5: Overview of the proposed BMG-Q framework for ride-pooling vehicles dispatch. For every decision round, initially users submit orders through a mobile application, and the system updates and sorts these orders alongside vehicle information. Taking into account long-term uncertainties including the intricate interdependence of agents, the GATDDQN network evaluates and dynamically assigns orders to suitable vehicles using ILP. After assignments, vehicle agents execute their actions, with experiences collected for subsequent learning phases of the GATDDQN network. Concurrently, the Open Street Routing Machine updates routes and estimated times of arrival, which are communicated back to the central platform. (Part of icons and map are from GoogleImagesGoogleMaps).

Theorems & Definitions (2)

• Remark 1
• Remark 2