Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll, Philip Schörner, J. Marius Zöllner

Abstract

Autonomous driving involves multiple, often conflicting objectives such as safety, efficiency, and comfort. In reinforcement learning (RL), these objectives are typically combined through weighted summation, which collapses their relative priorities and often yields policies that violate safety-critical constraints. To overcome this limitation, we introduce the Preordered Multi-Objective MDP (Pr-MOMDP), which augments standard MOMDPs with a preorder over reward components. This structure enables reasoning about actions with respect to a hierarchy of objectives rather than a scalar signal. To make this structure actionable, we extend distributional RL with a novel pairwise comparison metric, Quantile Dominance (QD), that evaluates action return distributions without reducing them into a single statistic. Building on QD, we propose an algorithm for extracting optimal subsets, the subset of actions that remain non-dominated under each objective, which allows precedence information to shape both decision-making and training targets. Our framework is instantiated with Implicit Quantile Networks (IQN), establishing a concrete implementation while preserving compatibility with a broad class of distributional RL methods. Experiments in Carla show improved success rates, fewer collisions and off-road events, and deliver statistically more robust policies than IQN and ensemble-IQN baselines. By ensuring policies respect rewards preorder, our work advances safer, more reliable autonomous driving systems.

Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

Abstract

Autonomous driving involves multiple, often conflicting objectives such as safety, efficiency, and comfort. In reinforcement learning (RL), these objectives are typically combined through weighted summation, which collapses their relative priorities and often yields policies that violate safety-critical constraints. To overcome this limitation, we introduce the Preordered Multi-Objective MDP (Pr-MOMDP), which augments standard MOMDPs with a preorder over reward components. This structure enables reasoning about actions with respect to a hierarchy of objectives rather than a scalar signal. To make this structure actionable, we extend distributional RL with a novel pairwise comparison metric, Quantile Dominance (QD), that evaluates action return distributions without reducing them into a single statistic. Building on QD, we propose an algorithm for extracting optimal subsets, the subset of actions that remain non-dominated under each objective, which allows precedence information to shape both decision-making and training targets. Our framework is instantiated with Implicit Quantile Networks (IQN), establishing a concrete implementation while preserving compatibility with a broad class of distributional RL methods. Experiments in Carla show improved success rates, fewer collisions and off-road events, and deliver statistically more robust policies than IQN and ensemble-IQN baselines. By ensuring policies respect rewards preorder, our work advances safer, more reliable autonomous driving systems.
Paper Structure (20 sections, 6 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 20 sections, 6 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. related work
  3. Reward Design in Autonomous Driving
  4. Multi-Objective RL and Hierarchical Rewards.
  5. Methodology
  6. Problem Formulation
  7. Action Relations from Rewards Preorder
  8. Preorder-guided Action Selection
  9. Agent Architecture
  10. Value Functions Estimation
  11. Distribution-aware Pairwise Comparison
  12. Preorder Traversal and Action Selection
  13. Preorder Informed Training and Inference
  14. experimental Setup
  15. RL Agent Description
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Examples of lexicographic and partial order rewards
  • Figure 2: Comparison of two classical architectures (a, b) with our Pr-IQN approach (c), shown during inference given observations $o_t$. Information bottlenecks are highlighted in red and novel components for full information utilization in green. Compared to classical approaches, Pr-IQN leverages distributions $Z^{r_n}$ to select actions that respect a given preorder.
  • Figure 3: The reward hierarchy with Safety as the highest priority, followed by Risk, Lane Keeping, Progress, and Comfort. This ordering guides the agent’s decision-making to emphasize safety while balancing other objectives.
  • Figure 4: Probability of improvement agarwal2021deep, quantifying the likelihood that an algorithm X (the left column) outperforms algorithm Y (the right column).
  • Figure 5: Interquartile mean (IQM) and optimality gap agarwal2021deep, quantifying the statistical stability of a policy.