Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with Distributed MPC for Fuel-Efficient Platoon Control with Discrete Gear Transitions

Samuel Mallick, Gianpietro Battocletti, Dimitris Boskos, Azita Dabiri, Bart De Schutter

TL;DR

This paper tackles fuel-efficient platooning with discrete gear transitions by introducing a learning-augmented distributed MPC. A recurrent neural network-based gear-shift policy, trained with reinforcement learning, fixes gear sequences across the MPC horizon, reducing online computation from solving MINLPs to solving NLPs while preserving co-optimization of speed and gear. The authors establish feasible gear schedules, decouple learning from platoon size via shifted-state inputs, and demonstrate through highway simulations that the LC-2 controller achieves near-MINLP performance with orders-of-magnitude faster computation. The approach offers scalable, real-time platoon control with robust transfer from single-vehicle training to multi-vehicle deployment, holding promise for practical fuel-efficiency gains in autonomous highways.

Abstract

Cooperative control of groups of autonomous vehicles (AVs), i.e., platoons, is a promising direction to improving the efficiency of autonomous transportation systems. In this context, distributed co-optimization of both vehicle speed and gear position can offer benefits for fuel-efficient driving. To this end, model predictive control (MPC) is a popular approach, optimizing the speed and gear-shift schedule while explicitly considering the vehicles' dynamics over a prediction window. However, optimization over both the vehicles' continuous dynamics and discrete gear positions is computationally intensive, and may require overly long sample times or high-end hardware for real-time implementation. This work proposes a reinforcement learning (RL)-based distributed MPC approach to address this issue. For each vehicle in the platoon, a policy is trained to select and fix the gear positions across the prediction window of a local MPC controller, leaving a significantly simpler continuous optimization problem to be solved as part of a distributed MPC scheme. In order to reduce the computational cost of training and facilitate the scalability of the proposed approach to large platoons, the policies are parameterized such that the emergent multi-agent RL problem can be decoupled into single-agent learning tasks. In addition, a recurrent neural-network (RNN) architecture is proposed for the gear selection policy, such that the learning is scalable even as the number of possible gear-shift schedules grows exponentially with the MPC prediction horizon. In highway-driving simulations, the proposed approach is shown to have a significantly lower computation burden and a comparable performance in terms of fuel-efficient platoon control, with respect to pure MPC-based co-optimization.

Reinforcement Learning with Distributed MPC for Fuel-Efficient Platoon Control with Discrete Gear Transitions

TL;DR

This paper tackles fuel-efficient platooning with discrete gear transitions by introducing a learning-augmented distributed MPC. A recurrent neural network-based gear-shift policy, trained with reinforcement learning, fixes gear sequences across the MPC horizon, reducing online computation from solving MINLPs to solving NLPs while preserving co-optimization of speed and gear. The authors establish feasible gear schedules, decouple learning from platoon size via shifted-state inputs, and demonstrate through highway simulations that the LC-2 controller achieves near-MINLP performance with orders-of-magnitude faster computation. The approach offers scalable, real-time platoon control with robust transfer from single-vehicle training to multi-vehicle deployment, holding promise for practical fuel-efficiency gains in autonomous highways.

Abstract

Cooperative control of groups of autonomous vehicles (AVs), i.e., platoons, is a promising direction to improving the efficiency of autonomous transportation systems. In this context, distributed co-optimization of both vehicle speed and gear position can offer benefits for fuel-efficient driving. To this end, model predictive control (MPC) is a popular approach, optimizing the speed and gear-shift schedule while explicitly considering the vehicles' dynamics over a prediction window. However, optimization over both the vehicles' continuous dynamics and discrete gear positions is computationally intensive, and may require overly long sample times or high-end hardware for real-time implementation. This work proposes a reinforcement learning (RL)-based distributed MPC approach to address this issue. For each vehicle in the platoon, a policy is trained to select and fix the gear positions across the prediction window of a local MPC controller, leaving a significantly simpler continuous optimization problem to be solved as part of a distributed MPC scheme. In order to reduce the computational cost of training and facilitate the scalability of the proposed approach to large platoons, the policies are parameterized such that the emergent multi-agent RL problem can be decoupled into single-agent learning tasks. In addition, a recurrent neural-network (RNN) architecture is proposed for the gear selection policy, such that the learning is scalable even as the number of possible gear-shift schedules grows exponentially with the MPC prediction horizon. In highway-driving simulations, the proposed approach is shown to have a significantly lower computation burden and a comparable performance in terms of fuel-efficient platoon control, with respect to pure MPC-based co-optimization.
Paper Structure (31 sections, 1 theorem, 60 equations, 8 figures, 3 tables, 2 algorithms)

This paper contains 31 sections, 1 theorem, 60 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting and Background
  3. Preliminaries and notation
  4. Reinforcement Learning and Deep Q-Network Algorithm
  5. AV model
  6. Problem setting
  7. Distributed Mixed-Integer Nonlinear MPC
  8. Distributed Parameterized Nonlinear MPC
  9. Feasible gear-shift schedules
  10. Gear-Shift Schedule Policy
  11. Design of wo
  12. Parameterization
  13. Architecture
  14. Training of wo
  15. Supervised versus reinforcement learning
  16. ...and 16 more sections

Key Result

Proposition 1

mallick2025learning Assume that, for $j \in \{1,\dots,j_\mathrm{max}\}$ and for all $v \in \Omega(j)$, there exist $T$ and $F$ such that $T_\text{min} \leq T \leq T_\text{max}$, $F_\text{max} \leq F \leq F_\text{max}$, and Then, for a state $x_i(k)$ such that $v_\mathrm{min} \leq x_i^{[2]}(k) \leq v_\mathrm{max}$, and a gear-shift sequence $\textbf{j}_i(k) = \rho_{\mathrm{const}, \phi}(x(k))$, pr

Figures (8)

  • Figure 1: Recurrent NN structure. The maps $\psi$ and $\eta$ are input and output transformations, respectively.
  • Figure 2: RL training setup. At the end of the training process, the policy that gets deployed is $\varpi_\theta$, indicated by the dashed box.
  • Figure 3: Performance metrics over the training stage 1. The plots show mean (solid) and standard deviation (shaded) of the moving average of the metrics for a moving window of 10000 steps.
  • Figure 4: Representative trajectories during training stage 1. The red dashed line in the second row represents $x^{[2]}_\mathrm{ref}$.
  • Figure 5: Performance metrics over the training stage 2. The plots show mean (solid) and standard deviation (shaded) of the moving average of the metrics for a moving window of 1000 steps.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Remark 1
  • Proposition 1
  • Remark 2