Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Objective Reinforcement Learning for Efficient Tactical Decision Making for Trucks in Highway Traffic

Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

TL;DR

This paper tackles the problem of balancing safety, time efficiency, and energy efficiency for autonomous trucks in highway traffic. It proposes a multi-objective reinforcement learning framework based on Proximal Policy Optimization (MOPPO) combined with Generalized Policy Improvement with Linear Support (GPI-LS) to learn a continuous set of Pareto-optimal policies, preserving objective structure through weight-conditioned components. Key contributions include extending GPI-LS to a policy-gradient setting, introducing weight-conditioned multi-objective PPO with action masking and a multi-dimensional critic, and implementing a safety-filtered lane-change mechanism. Experimental results on a high-fidelity SUMO-based highway simulator show that the approach efficiently approximates the Convex Coverage Set, enabling dynamic, preference-aware policy selection with robust performance across traffic densities, and the authors release the framework as open source for reproducibility and further research.

Abstract

Balancing safety, efficiency, and operational costs in highway driving poses a challenging decision-making problem for heavy-duty vehicles. A central difficulty is that conventional scalar reward formulations, obtained by aggregating these competing objectives, often obscure the structure of their trade-offs. We present a Proximal Policy Optimization based multi-objective reinforcement learning framework that learns a continuous set of policies explicitly representing these trade-offs and evaluates it on a scalable simulation platform for tactical decision making in trucks. The proposed approach learns a continuous set of Pareto-optimal policies that capture the trade-offs among three conflicting objectives: safety, quantified in terms of collisions and successful completion; energy efficiency and time efficiency, quantified using energy cost and driver cost, respectively. The resulting Pareto frontier is smooth and interpretable, enabling flexibility in choosing driving behavior along different conflicting objectives. This framework allows seamless transitions between different driving policies without retraining, yielding a robust and adaptive decision-making strategy for autonomous trucking applications.

Multi-Objective Reinforcement Learning for Efficient Tactical Decision Making for Trucks in Highway Traffic

TL;DR

This paper tackles the problem of balancing safety, time efficiency, and energy efficiency for autonomous trucks in highway traffic. It proposes a multi-objective reinforcement learning framework based on Proximal Policy Optimization (MOPPO) combined with Generalized Policy Improvement with Linear Support (GPI-LS) to learn a continuous set of Pareto-optimal policies, preserving objective structure through weight-conditioned components. Key contributions include extending GPI-LS to a policy-gradient setting, introducing weight-conditioned multi-objective PPO with action masking and a multi-dimensional critic, and implementing a safety-filtered lane-change mechanism. Experimental results on a high-fidelity SUMO-based highway simulator show that the approach efficiently approximates the Convex Coverage Set, enabling dynamic, preference-aware policy selection with robust performance across traffic densities, and the authors release the framework as open source for reproducibility and further research.

Abstract

Balancing safety, efficiency, and operational costs in highway driving poses a challenging decision-making problem for heavy-duty vehicles. A central difficulty is that conventional scalar reward formulations, obtained by aggregating these competing objectives, often obscure the structure of their trade-offs. We present a Proximal Policy Optimization based multi-objective reinforcement learning framework that learns a continuous set of policies explicitly representing these trade-offs and evaluates it on a scalable simulation platform for tactical decision making in trucks. The proposed approach learns a continuous set of Pareto-optimal policies that capture the trade-offs among three conflicting objectives: safety, quantified in terms of collisions and successful completion; energy efficiency and time efficiency, quantified using energy cost and driver cost, respectively. The resulting Pareto frontier is smooth and interpretable, enabling flexibility in choosing driving behavior along different conflicting objectives. This framework allows seamless transitions between different driving policies without retraining, yielding a robust and adaptive decision-making strategy for autonomous trucking applications.
Paper Structure (16 sections, 15 equations, 9 figures, 5 tables, 2 algorithms)

This paper contains 16 sections, 15 equations, 9 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Decision Making in Traffic Environment
  4. Multi-Objective Reinforcement Learning
  5. Reinforcement Learning Environment
  6. Methodology
  7. GPI-Based Multi-Objective Reinforcement Learning
  8. Multi-Objective Proximal Policy Optimization (MOPPO)
  9. Safety Filter for Lane Changes via Action Masking
  10. Experimental Results
  11. Conclusion
  12. Multi-Objective Reinforcement Learning Literature
  13. Traffic Modeling
  14. Detailed Architecture
  15. Reward Computation
  16. ...and 1 more sections

Figures (9)

  • Figure 1: Multi-Objective PPO Framework.
  • Figure 2: Pareto Front showing the trade off between driver cost and energy cost in three different traffic settings, along with the success rate.
  • Figure 3: Comparison of average speed and cost values from Pareto-optimal policies with $100\%$ success rate with the analytical values.
  • Figure 4: Simulated traffic environment in SUMO with the illustration of moving window.
  • Figure 5: Overview of the architecture adapted from pathare2025tactical, extended for multi-objective learning
  • ...and 4 more figures