Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving

Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, Jun Wang

TL;DR

SMARTS addresses the challenge of learning through interaction with diverse road users by providing a scalable, open-source simulation platform tailored for multi-agent reinforcement learning in autonomous driving. It introduces the Social Agent Zoo and a bubble-based architecture to bootstrap realistic, diverse interactions, supported by a domain-specific language and a suite of providers for traffic, physics, and motion planning. The framework integrates with major MARL libraries and offers a comprehensive benchmarking suite with AD-specific metrics, enabling scalable training and evaluation across multiple scenarios. Empirical results across three driving scenarios and seven MARL algorithms demonstrate the platform’s ability to characterize performance, robustness, and behavior diversity, highlighting MADDPG’s strengths in complex interactions and illustrating the platform’s practical impact for advancing MARL-driven autonomous driving research.

Abstract

Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, and use of diverse behavior models of road users. These are in turn used to create increasingly more realistic and diverse interactions that enable deeper and broader research on multi-agent interaction. In this paper, we describe the design goals of SMARTS, explain its basic architecture and its key features, and illustrate its use through concrete multi-agent experiments on interactive scenarios. We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving. Our code is available at https://github.com/huawei-noah/SMARTS.

SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving

TL;DR

SMARTS addresses the challenge of learning through interaction with diverse road users by providing a scalable, open-source simulation platform tailored for multi-agent reinforcement learning in autonomous driving. It introduces the Social Agent Zoo and a bubble-based architecture to bootstrap realistic, diverse interactions, supported by a domain-specific language and a suite of providers for traffic, physics, and motion planning. The framework integrates with major MARL libraries and offers a comprehensive benchmarking suite with AD-specific metrics, enabling scalable training and evaluation across multiple scenarios. Empirical results across three driving scenarios and seven MARL algorithms demonstrate the platform’s ability to characterize performance, robustness, and behavior diversity, highlighting MADDPG’s strengths in complex interactions and illustrating the platform’s practical impact for advancing MARL-driven autonomous driving research.

Abstract

Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, and use of diverse behavior models of road users. These are in turn used to create increasingly more realistic and diverse interactions that enable deeper and broader research on multi-agent interaction. In this paper, we describe the design goals of SMARTS, explain its basic architecture and its key features, and illustrate its use through concrete multi-agent experiments on interactive scenarios. We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving. Our code is available at https://github.com/huawei-noah/SMARTS.

Paper Structure

This paper contains 19 sections, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Design Goals of the SMARTS Platform
  3. Architecture and Implementation
  4. Support for Multi-Agent Reinforcement Learning
  5. Experiments & Results
  6. Observation.
  7. Action.
  8. Reward.
  9. Conclusion
  10. Survey of AD Related Simulators
  11. SMARTS Simulations
  12. Controllers and Action Spaces
  13. Algorithm Library
  14. Experiment Results
  15. User Examples
  16. ...and 4 more sections

Figures (11)

  • Figure 1: The double merge scenario.
  • Figure 2: Bootstrapping interaction realism and diversity.
  • Figure 3: SMARTS Architecture. Interaction scenarios are defined with a domain-specific-language (DSL). Social agents are instantiated from the Social Agent Zoo. Orange vehicles are controlled by learning agents, dark blue vehicles by social agents, light blue ones by traffic provider. All providers and agents in principle run in their own processes, possibly remotely.
  • Figure 4: Bubble. Inside A Zone, social vehicles are controlled by the traffic provider; inside B Zone, by agents from the Social Agent Zoo.
  • Figure 5: Scenarios of driving interaction specifiable in SMARTS
  • ...and 6 more figures