Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Progress and Safety: A Novel Risk-Aware Objective for RL in Autonomous Driving

Ahmed Abouelazm, Jonas Michel, Helen Gremmelmaier, Tim Joseph, Philip Schörner, J. Marius Zöllner

TL;DR

The approach decreases collision rates by 21% on average compared to baseline rewards and consistently surpasses them in route progress and cumulative reward, demonstrating its capability to promote safer driving behaviors while maintaining high-performance levels.

Abstract

Reinforcement Learning (RL) is a promising approach for achieving autonomous driving due to robust decision-making capabilities. RL learns a driving policy through trial and error in traffic scenarios, guided by a reward function that combines the driving objectives. The design of such reward function has received insufficient attention, yielding ill-defined rewards with various pitfalls. Safety, in particular, has long been regarded only as a penalty for collisions. This leaves the risks associated with actions leading up to a collision unaddressed, limiting the applicability of RL in real-world scenarios. To address these shortcomings, our work focuses on enhancing the reward formulation by defining a set of driving objectives and structuring them hierarchically. Furthermore, we discuss the formulation of these objectives in a normalized manner to transparently determine their contribution to the overall reward. Additionally, we introduce a novel risk-aware objective for various driving interactions based on a two-dimensional ellipsoid function and an extension of Responsibility-Sensitive Safety (RSS) concepts. We evaluate the efficacy of our proposed reward in unsignalized intersection scenarios with varying traffic densities. The approach decreases collision rates by 21\% on average compared to baseline rewards and consistently surpasses them in route progress and cumulative reward, demonstrating its capability to promote safer driving behaviors while maintaining high-performance levels.

Balancing Progress and Safety: A Novel Risk-Aware Objective for RL in Autonomous Driving

TL;DR

The approach decreases collision rates by 21% on average compared to baseline rewards and consistently surpasses them in route progress and cumulative reward, demonstrating its capability to promote safer driving behaviors while maintaining high-performance levels.

Abstract

Reinforcement Learning (RL) is a promising approach for achieving autonomous driving due to robust decision-making capabilities. RL learns a driving policy through trial and error in traffic scenarios, guided by a reward function that combines the driving objectives. The design of such reward function has received insufficient attention, yielding ill-defined rewards with various pitfalls. Safety, in particular, has long been regarded only as a penalty for collisions. This leaves the risks associated with actions leading up to a collision unaddressed, limiting the applicability of RL in real-world scenarios. To address these shortcomings, our work focuses on enhancing the reward formulation by defining a set of driving objectives and structuring them hierarchically. Furthermore, we discuss the formulation of these objectives in a normalized manner to transparently determine their contribution to the overall reward. Additionally, we introduce a novel risk-aware objective for various driving interactions based on a two-dimensional ellipsoid function and an extension of Responsibility-Sensitive Safety (RSS) concepts. We evaluate the efficacy of our proposed reward in unsignalized intersection scenarios with varying traffic densities. The approach decreases collision rates by 21\% on average compared to baseline rewards and consistently surpasses them in route progress and cumulative reward, demonstrating its capability to promote safer driving behaviors while maintaining high-performance levels.
Paper Structure (13 sections, 17 equations, 4 figures, 2 tables)

This paper contains 13 sections, 17 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RL for Autonomous Driving
  4. Reward Functions for Autonomous Driving
  5. Method
  6. Reward Structure and Weights Assignment
  7. Safe and Risk-aware Reward Design
  8. Experimental Setup
  9. RL Agent Description
  10. Traffic Scenarios
  11. Baselines and Evaluation Metrics
  12. Evaluation
  13. Conclusion

Figures (4)

  • Figure 1: Example of irrational behavior in RL agents caused by sparse safety objective formulation and poor handling of the conflict between safety and progress objectives misdesign.
  • Figure 2: Representation of the driving reward as a directed graph, where the objective level indicates its priority.
  • Figure 3: The risk-aware objective considers different interactions between traffic participants. The RL agent is highlighted in green, and Non-Player Characters (NPCs) and obstacles are in blue.
  • Figure 4: Visualization of one-dimensional ellipsoid penalty based on the distance between RL agent (green) and another vehicle (blue). The risk field ranges from 1 to $\epsilon$, where $\epsilon$ is close to zero.