Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximate Multiagent Reinforcement Learning for On-Demand Urban Mobility Problem on a Large Map (extended version)

Daniel Garces, Sushmita Bhattacharya, Dimitri Bertsekas, Stephanie Gil

TL;DR

This work tackles large-scale autonomous taxi routing under unknown future demand by introducing a two-phase approximate rollout algorithm. The map is partitioned into sectors and taxis are planned across sectors with a high-level cross-sector planner, while a low-level per-sector rollout uses instantaneous assignment with reassignment (IA-RA) as the base policy, exploiting sector parallelism to dramatically reduce computation. The authors prove a sufficient condition on fleet size $m$ for stability of the IA-RA base policy via $m \ge E[\eta] D_{\max}$ and, under independent demand, a necessary condition $m > E[\eta] D_{\min}$ for asymptotic stability, where $D_{\max}$ and $D_{\min}$ involve distance expectations and the Wasserstein distance between pickup and drop-off distributions. Numerical results on San Francisco data show that the two-phase approach achieves stability and near-parallel performance to a full-map rollout while delivering substantial runtime savings, and scales favorably with fleet size. Overall, the method enables scalable, stable, near-optimal on-demand urban mobility planning on large maps with tractable computation.

Abstract

In this paper, we focus on the autonomous multiagent taxi routing problem for a large urban environment where the location and number of future ride requests are unknown a-priori, but can be estimated by an empirical distribution. Recent theory has shown that a rollout algorithm with a stable base policy produces a near-optimal stable policy. In the routing setting, a policy is stable if its execution keeps the number of outstanding requests uniformly bounded over time. Although, rollout-based approaches are well-suited for learning cooperative multiagent policies with considerations for future demand, applying such methods to a large urban environment can be computationally expensive due to the large number of taxis required for stability. In this paper, we aim to address the computational bottleneck of multiagent rollout by proposing an approximate multiagent rollout-based two phase algorithm that reduces computational costs, while still achieving a stable near-optimal policy. Our approach partitions the graph into sectors based on the predicted demand and the maximum number of taxis that can run sequentially given the user's computational resources. The algorithm then applies instantaneous assignment (IA) for re-balancing taxis across sectors and a sector-wide multiagent rollout algorithm that is executed in parallel for each sector. We provide two main theoretical results: 1) characterize the number of taxis $m$ that is sufficient for IA to be stable; 2) derive a necessary condition on $m$ to maintain stability for IA as time goes to infinity. Our numerical results show that our approach achieves stability for an $m$ that satisfies the theoretical conditions. We also empirically demonstrate that our proposed two phase algorithm has equivalent performance to the one-at-a-time rollout over the entire map, but with significantly lower runtimes.

Approximate Multiagent Reinforcement Learning for On-Demand Urban Mobility Problem on a Large Map (extended version)

TL;DR

This work tackles large-scale autonomous taxi routing under unknown future demand by introducing a two-phase approximate rollout algorithm. The map is partitioned into sectors and taxis are planned across sectors with a high-level cross-sector planner, while a low-level per-sector rollout uses instantaneous assignment with reassignment (IA-RA) as the base policy, exploiting sector parallelism to dramatically reduce computation. The authors prove a sufficient condition on fleet size for stability of the IA-RA base policy via and, under independent demand, a necessary condition for asymptotic stability, where and involve distance expectations and the Wasserstein distance between pickup and drop-off distributions. Numerical results on San Francisco data show that the two-phase approach achieves stability and near-parallel performance to a full-map rollout while delivering substantial runtime savings, and scales favorably with fleet size. Overall, the method enables scalable, stable, near-optimal on-demand urban mobility planning on large maps with tractable computation.

Abstract

In this paper, we focus on the autonomous multiagent taxi routing problem for a large urban environment where the location and number of future ride requests are unknown a-priori, but can be estimated by an empirical distribution. Recent theory has shown that a rollout algorithm with a stable base policy produces a near-optimal stable policy. In the routing setting, a policy is stable if its execution keeps the number of outstanding requests uniformly bounded over time. Although, rollout-based approaches are well-suited for learning cooperative multiagent policies with considerations for future demand, applying such methods to a large urban environment can be computationally expensive due to the large number of taxis required for stability. In this paper, we aim to address the computational bottleneck of multiagent rollout by proposing an approximate multiagent rollout-based two phase algorithm that reduces computational costs, while still achieving a stable near-optimal policy. Our approach partitions the graph into sectors based on the predicted demand and the maximum number of taxis that can run sequentially given the user's computational resources. The algorithm then applies instantaneous assignment (IA) for re-balancing taxis across sectors and a sector-wide multiagent rollout algorithm that is executed in parallel for each sector. We provide two main theoretical results: 1) characterize the number of taxis that is sufficient for IA to be stable; 2) derive a necessary condition on to maintain stability for IA as time goes to infinity. Our numerical results show that our approach achieves stability for an that satisfies the theoretical conditions. We also empirically demonstrate that our proposed two phase algorithm has equivalent performance to the one-at-a-time rollout over the entire map, but with significantly lower runtimes.
Paper Structure (21 sections, 3 theorems, 26 equations, 5 figures, 3 algorithms)

This paper contains 21 sections, 3 theorems, 26 equations, 5 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Environment
  4. Requests
  5. State and control space
  6. Stability of a policy
  7. Challenges of a large scale multi-agent problem
  8. Approximation algorithm for multiagent rollout
  9. Theoretical Results
  10. Sufficient condition for stability of $\pi_{\text{base}}$
  11. Necessary condition for stability of $\pi_{\text{base}}$
  12. Numerical studies
  13. Experimental Setup
  14. Estimating probability distributions
  15. Calculated values for theoretical results
  16. ...and 6 more sections

Key Result

Lemma 1

Let the random variable $l_{\text{rand}}$ with support $V$ represent the location of a random taxi that gets assigned to a request after that taxi has previously served a different request. Define $D_{\text{max}} \triangleq \max\{E[d(\xi, \rho)], E[d(l_{\text{rand}}, \rho)]\} + E[d(\rho, \delta)]$. then the policy associated with a random instantaneous assignment of taxis to requests, $\hat{\pi}$

Figures (5)

  • Figure 1: Street network used in numerical experiments
  • Figure 2: Our two phased approach executed on a map with 3 sectors.
  • Figure 3: Total wait time over all requests of our two-phase approach and the benchmarks.
  • Figure 4: Execution time comparison between our two-phase approach and the benchmarks.
  • Figure 5: Stability of IA-RA and two-phase policy in terms of the means (lines) and standard deviations (shaded regions) of the number of outstanding requests.

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof