Prediction-aware and Reinforcement Learning based Altruistic Cooperative Driving

TL;DR

This work addresses safe, efficient AV navigation in mixed autonomy by enabling prediction of HV behavior and integrating prediction into socially-aware reinforcement learning. It introduces a Hybrid Predictive Network (HPN) to forecast multi-step future observations, which feed a Safe Value Function Network (VFN) that optimizes social utility within a POSG framework, constrained by a safety prioritizer. Key contributions include formalizing altruistic cooperative driving as POSG, developing a prediction chain that supplies the VFN with forward-looking information, and demonstrating improved safety and efficiency across diverse driving scenarios with HVs of varying behaviors. The approach advances practical AV-HV interaction by enabling proactive, interpretable, and safer decision-making in complex traffic mixes.

Abstract

Autonomous vehicle (AV) navigation in the presence of Human-driven vehicles (HVs) is challenging, as HVs continuously update their policies in response to AVs. In order to navigate safely in the presence of complex AV-HV social interactions, the AVs must learn to predict these changes. Humans are capable of navigating such challenging social interaction settings because of their intrinsic knowledge about other agents behaviors and use that to forecast what might happen in the future. Inspired by humans, we provide our AVs the capability of anticipating future states and leveraging prediction in a cooperative reinforcement learning (RL) decision-making framework, to improve safety and robustness. In this paper, we propose an integration of two essential and earlier-presented components of AVs: social navigation and prediction. We formulate the AV decision-making process as a RL problem and seek to obtain optimal policies that produce socially beneficial results utilizing a prediction-aware planning and social-aware optimization RL framework. We also propose a Hybrid Predictive Network (HPN) that anticipates future observations. The HPN is used in a multi-step prediction chain to compute a window of predicted future observations to be used by the value function network (VFN). Finally, a safe VFN is trained to optimize a social utility using a sequence of previous and predicted observations, and a safety prioritizer is used to leverage the interpretable kinematic predictions to mask the unsafe actions, constraining the RL policy. We compare our prediction-aware AV to state-of-the-art solutions and demonstrate performance improvements in terms of efficiency and safety in multiple simulated scenarios.

Figures (10)

• Figure 1: It is crucial for AV being capable of anticipating future situations of other vehicles using spatial and temporal information. Decision-making by AVs can be enhanced by anticipating the intentions of other agents, which is crucial in complex scenarios and safety-critical situations. The figure depicts an AV (in green) and HVs (in blue) with corresponding predictions within the AVs perception window.
• Figure 2: An overview of our prediction-aware and social-aware cooperative driving approach for Multi-agent Cooperative Reinforcement Learning to improve the safety of CAVs. The proposed decentralized RL architecture employs an HPN, which gives AVs the ability to anticipate future states, which is used by the VFN to optimize for social utility and by the safety prioritizer to avoid high collision potential actions.
• Figure 3: Architecture of the hybrid predictive network (HPN) for one prediction step. Using HPN in a chain will give AVs the ability to predict future observations.
• Figure 4: Multi-step prediction chain and Safe Value Function Network (VFN). The prediction chain is a multi-step prediction chain that uses the HPN in a chain to produce a set of $M$ future hypotheses. The VFN is a 3D CNN that acts as a function approximator, it uses temporal information and prediction to improve decision-making.
• Figure 5: Kinematic prediction baseline comparison in terms of position error ($PE$) in meters.