Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Probabilistic Ensembles with Trajectory Sampling for Connected Autonomous Vehicles

Ruoqi Wen, Jiahao Huang, Rongpeng Li, Guoru Ding, Zhifeng Zhao

TL;DR

This work tackles sample-efficient decision-making for multiple connected autonomous vehicles under limited communications by introducing MA-PETS, a fully decentralized multi-agent model-based RL method. MA-PETS extends PETS with Probabilistic Ensemble dynamics and trajectory-sampling MPC, enabling joint planning with data shared among neighboring CAVs. The authors derive a group regret bound using optimistic MDPs and a clique-cover-based analysis to show sub-linear growth in regret with the number of agents and communication range. Empirical validation on SMARTS demonstrates superior sample efficiency and robust performance compared to several MARL baselines, highlighting the value of coordinated information exchange and trajectory-based planning in complex traffic scenarios.

Abstract

Autonomous Vehicles (AVs) have attracted significant attention in recent years and Reinforcement Learning (RL) has shown remarkable performance in improving the autonomy of vehicles. In that regard, the widely adopted Model-Free RL (MFRL) promises to solve decision-making tasks in connected AVs (CAVs), contingent on the readiness of a significant amount of data samples for training. Nevertheless, it might be infeasible in practice and possibly lead to learning instability. In contrast, Model-Based RL (MBRL) manifests itself in sample-efficient learning, but the asymptotic performance of MBRL might lag behind the state-of-the-art MFRL algorithms. Furthermore, most studies for CAVs are limited to the decision-making of a single AV only, thus underscoring the performance due to the absence of communications. In this study, we try to address the decision-making problem of multiple CAVs with limited communications and propose a decentralized Multi-Agent Probabilistic Ensembles with Trajectory Sampling algorithm MA-PETS. In particular, in order to better capture the uncertainty of the unknown environment, MA-PETS leverages Probabilistic Ensemble (PE) neural networks to learn from communicated samples among neighboring CAVs. Afterwards, MA-PETS capably develops Trajectory Sampling (TS)-based model-predictive control for decision-making. On this basis, we derive the multi-agent group regret bound affected by the number of agents within the communication range and mathematically validate that incorporating effective information exchange among agents into the multi-agent learning scheme contributes to reducing the group regret bound in the worst case. Finally, we empirically demonstrate the superiority of MA-PETS in terms of the sample efficiency comparable to MFBL.

Multi-Agent Probabilistic Ensembles with Trajectory Sampling for Connected Autonomous Vehicles

TL;DR

This work tackles sample-efficient decision-making for multiple connected autonomous vehicles under limited communications by introducing MA-PETS, a fully decentralized multi-agent model-based RL method. MA-PETS extends PETS with Probabilistic Ensemble dynamics and trajectory-sampling MPC, enabling joint planning with data shared among neighboring CAVs. The authors derive a group regret bound using optimistic MDPs and a clique-cover-based analysis to show sub-linear growth in regret with the number of agents and communication range. Empirical validation on SMARTS demonstrates superior sample efficiency and robust performance compared to several MARL baselines, highlighting the value of coordinated information exchange and trajectory-based planning in complex traffic scenarios.

Abstract

Autonomous Vehicles (AVs) have attracted significant attention in recent years and Reinforcement Learning (RL) has shown remarkable performance in improving the autonomy of vehicles. In that regard, the widely adopted Model-Free RL (MFRL) promises to solve decision-making tasks in connected AVs (CAVs), contingent on the readiness of a significant amount of data samples for training. Nevertheless, it might be infeasible in practice and possibly lead to learning instability. In contrast, Model-Based RL (MBRL) manifests itself in sample-efficient learning, but the asymptotic performance of MBRL might lag behind the state-of-the-art MFRL algorithms. Furthermore, most studies for CAVs are limited to the decision-making of a single AV only, thus underscoring the performance due to the absence of communications. In this study, we try to address the decision-making problem of multiple CAVs with limited communications and propose a decentralized Multi-Agent Probabilistic Ensembles with Trajectory Sampling algorithm MA-PETS. In particular, in order to better capture the uncertainty of the unknown environment, MA-PETS leverages Probabilistic Ensemble (PE) neural networks to learn from communicated samples among neighboring CAVs. Afterwards, MA-PETS capably develops Trajectory Sampling (TS)-based model-predictive control for decision-making. On this basis, we derive the multi-agent group regret bound affected by the number of agents within the communication range and mathematically validate that incorporating effective information exchange among agents into the multi-agent learning scheme contributes to reducing the group regret bound in the worst case. Finally, we empirically demonstrate the superiority of MA-PETS in terms of the sample efficiency comparable to MFBL.
Paper Structure (22 sections, 10 theorems, 28 equations, 13 figures, 4 tables, 1 algorithm)

This paper contains 22 sections, 10 theorems, 28 equations, 13 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-Agent Reinforcement Learning of CAVs
  4. Model-Based Reinforcement Learning
  5. Regret bounds of Reinforcement Learning
  6. Preliminaries and System Model
  7. Preliminaries
  8. Parallel Markov decision process
  9. Model-Based Reinforcement Learning
  10. System Model
  11. The MA-PETS Algorithm
  12. Learning the Dynamics Model
  13. Planning and Control with Learned Dynamics
  14. Regret Bound of MA-PETS
  15. EVI for an Optimistic MDP
  16. ...and 7 more sections

Key Result

Lemma 1

Consider $M^{(i)}$ as a time-homogeneous and communicating MDP with a diameter of $D$. Let $\mathcal{R}^{\ast(i)}_T(M^{(i)})$ represent the optimal $T$-step reward, and $\rho^{\ast(i)}$ denote the optimal average reward under the reward function $\mathcal{R}^{(i)}$. It can be asserted that for any M

Figures (13)

  • Figure 1: The complex mixed urban traffic control scenario with unsignalized lane merging and intersections.
  • Figure 2: The illustration of the MA-PETS algorithm for CAVs.
  • Figure 3: The "Unprotected Intersection" scenario in the closed single-lane and multi-lane "Figure Eight" loop for Simulations. (a) presents an aerial view of the "Figure Eight" loop, while (b) and (c) provide the regional enlarged view of the single-lane and multi-lane "Unprotected Intersection", respectively.
  • Figure 4: Comparison of utility in the single-lane "Unprotected Intersection" scenario.
  • Figure 5: Real-time utility comparison for each step of the $10$-th episode under the single-lane "Unprotected Intersection" scenario.
  • ...and 8 more figures

Theorems & Definitions (11)

  • Lemma 1: Lemma 10 of gajane_variational_2019
  • Lemma 2: Eqs. (7)-(8) of UCRL2
  • Lemma 3: Regret with Failing Confidence Intervals, Eqs. (13) of ortner_online_2012
  • Lemma 4: Regret with $M^{(i)} \in \mathcal{M}^{(i)}_k$, Sec. 5.2 of Ortner2014SelectingNA
  • Corollary 1
  • Lemma 5: Eqs. (16)-(17) of UCRL2
  • Lemma 6: Eq. (6) of 9867146
  • Lemma 7: Theorem 1 of 9867146
  • Theorem 1
  • proof
  • ...and 1 more