Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic

TL;DR

<3-5 sentence high-level summary> The paper tackles the challenge of deploying reinforcement learning–based cooperative autonomous driving in mixed-autonomy traffic where human drivers exhibit heterogeneous and non-stationary behavior. It proposes a decentralized multi-agent reinforcement learning framework with a general social reward and a safety prioritizer to learn altruistic AV policies that are robust to diverse HV behaviors and scenarios. The approach combines a DDQN-based learning architecture, a 3D CNN observation encoder, and a safety mechanism to avoid high-risk actions, while modeling HVs with IDM/MOBIL dynamics and adaptable SVO weighting. Experimental results demonstrate improved safety and efficiency, effective domain adaptation across driving scenarios and HV styles, and faster transfer learning, highlighting the practical potential of safe, socially aware AVs in real-world traffic.

Abstract

Building autonomous vehicles (AVs) is a complex problem, but enabling them to operate in the real world where they will be surrounded by human-driven vehicles (HVs) is extremely challenging. Prior works have shown the possibilities of creating inter-agent cooperation between a group of AVs that follow a social utility. Such altruistic AVs can form alliances and affect the behavior of HVs to achieve socially desirable outcomes. We identify two major challenges in the co-existence of AVs and HVs. First, social preferences and individual traits of a given human driver, e.g., selflessness and aggressiveness are unknown to an AV, and it is almost impossible to infer them in real-time during a short AV-HV interaction. Second, contrary to AVs that are expected to follow a policy, HVs do not necessarily follow a stationary policy and therefore are extremely hard to predict. To alleviate the above-mentioned challenges, we formulate the mixed-autonomy problem as a multi-agent reinforcement learning (MARL) problem and propose a decentralized framework and reward function for training cooperative AVs. Our approach enables AVs to learn the decision-making of HVs implicitly from experience, optimizes for a social utility while prioritizing safety and allowing adaptability; robustifying altruistic AVs to different human behaviors and constraining them to a safe action space. Finally, we investigate the robustness, safety and sensitivity of AVs to various HVs behavioral traits and present the settings in which the AVs can learn cooperative policies that are adaptable to different situations.

Figures (9)

• Figure 1: An overview of our methodology for ensuring the safety and robustness of cooperative autonomous vehicles in interaction with human-driven vehicles.
• Figure 2: Highway merging and exiting scenarios in mixed traffic where the road is shared by AVs (green) and aggressive HVs (red). Altruistic AVs must learn to coordinate to allow for a safe and efficient merging/exiting while also being robust to different scenarios and behaviors and ensuring safety in decision-making.
• Figure 3: Deep MARL architecture with the safety prioritizer.
• Figure 4: Sensitivity analyses measured by altruistic performance gain (PG) of AVs, the more aggressiveness of the HVs, the higher the impact/gain of cooperation.
• Figure 5: Lateral and longitudinal sensitivity analyses, the altruistic performance gain (PG) increase in both lateral and longitudinal directions.