Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-armed Bandit for Stochastic Shortest Path in Mixed Autonomy

Yu Bai, Yiming Li, Xi Xiong

TL;DR

This work addresses real-time routing of autonomous vehicles in mixed-autonomy traffic under stochastic human-driven behavior by formulating the problem as a Stochastic Shortest Path task within an MDP. It introduces RTDP-UCB, a framework that injects UCB-based optimistic exploration into RTDP, and proves a path-level regret bound of $O\left(\sqrt{T \log T}\right)$ under asynchronous updates. The approach balances exploration and exploitation to achieve faster convergence to near-optimal routing policies, with empirical validation on a real Shanghai subnetwork showing superior convergence speed and computational efficiency compared to RTDP and VI baselines. The method holds promise for scalable, data-efficient routing in large, uncertain, mixed-autonomy networks with tangible practical impact for traffic efficiency and reliability.

Abstract

In mixed-autonomy traffic networks, autonomous vehicles (AVs) are required to make sequential routing decisions under uncertainty caused by dynamic and heterogeneous interactions with human-driven vehicles (HDVs). Early-stage greedy decisions made by AVs during interactions with the environment often result in insufficient exploration, leading to failures in discovering globally optimal strategies. The exploration-exploitation balancing mechanism inherent in multi-armed bandit (MAB) methods is well-suited for addressing such problems. Based on the Real-Time Dynamic Programming (RTDP) framework, we introduce the Upper Confidence Bound (UCB) exploration strategy from the MAB paradigm and propose a novel algorithm. We establish the path-level regret upper bound under the RTDP framework, which guarantees the worst-case convergence of the proposed algorithm. Extensive numerical experiments conducted on a real-world local road network in Shanghai demonstrate that the proposed algorithm effectively overcomes the failure of standard RTDP to converge to the optimal policy under highly stochastic environments. Moreover, compared to the standard Value Iteration (VI) framework, the RTDP-based framework demonstrates superior computational efficiency. Our results highlight the effectiveness of the proposed algorithm in routing within large-scale stochastic mixed-autonomy environments.

Multi-armed Bandit for Stochastic Shortest Path in Mixed Autonomy

TL;DR

This work addresses real-time routing of autonomous vehicles in mixed-autonomy traffic under stochastic human-driven behavior by formulating the problem as a Stochastic Shortest Path task within an MDP. It introduces RTDP-UCB, a framework that injects UCB-based optimistic exploration into RTDP, and proves a path-level regret bound of under asynchronous updates. The approach balances exploration and exploitation to achieve faster convergence to near-optimal routing policies, with empirical validation on a real Shanghai subnetwork showing superior convergence speed and computational efficiency compared to RTDP and VI baselines. The method holds promise for scalable, data-efficient routing in large, uncertain, mixed-autonomy networks with tangible practical impact for traffic efficiency and reliability.

Abstract

In mixed-autonomy traffic networks, autonomous vehicles (AVs) are required to make sequential routing decisions under uncertainty caused by dynamic and heterogeneous interactions with human-driven vehicles (HDVs). Early-stage greedy decisions made by AVs during interactions with the environment often result in insufficient exploration, leading to failures in discovering globally optimal strategies. The exploration-exploitation balancing mechanism inherent in multi-armed bandit (MAB) methods is well-suited for addressing such problems. Based on the Real-Time Dynamic Programming (RTDP) framework, we introduce the Upper Confidence Bound (UCB) exploration strategy from the MAB paradigm and propose a novel algorithm. We establish the path-level regret upper bound under the RTDP framework, which guarantees the worst-case convergence of the proposed algorithm. Extensive numerical experiments conducted on a real-world local road network in Shanghai demonstrate that the proposed algorithm effectively overcomes the failure of standard RTDP to converge to the optimal policy under highly stochastic environments. Moreover, compared to the standard Value Iteration (VI) framework, the RTDP-based framework demonstrates superior computational efficiency. Our results highlight the effectiveness of the proposed algorithm in routing within large-scale stochastic mixed-autonomy environments.
Paper Structure (7 sections, 4 theorems, 21 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 7 sections, 4 theorems, 21 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Modeling and Formulation
  3. Multi-Armed Bandit for adaptive routing in mixed-autonomy
  4. Price of Optimism
  5. Bellman Error
  6. Numerical Results
  7. CONCLUSIONS

Key Result

Proposition 1

Let $T > 0$ be the total number of episodes. Define $|\mathcal{E}|$ as the number of edges in the graph, $\mathcal{S}_{\text{rel}}$ as the set of states that are visited at least once across all episodes, and $L_{\max}$ as the maximum number of steps in all paths. The cumulative regret $R_T$ of the

Figures (4)

  • Figure 1: AVs Routing Scenario in Mixed-Autonomy Environments
  • Figure 2: Convergence of estimated state value $V(s_{o})$ over 300 iterations for four algorithms.
  • Figure 3: Cumulative and average regret over 300 episodes for the four evaluated algorithms.
  • Figure 4: Visualization of edge sampling during the simulation. The thickness of each edge represents the number of times it was sampled during the execution of the algorithm, indicating a certain degree of selection preference.

Theorems & Definitions (4)

  • Proposition 1
  • Lemma 1
  • Lemma 2
  • Lemma 3