Mixed-Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy Management

TL;DR

This work tackles mixed-integer optimal control (MIOC) by proposing TD3AQ, a hybrid-action reinforcement learning algorithm that jointly handles discrete and continuous actions within an actor-critic/Q-learning framework. By applying TD3AQ to a plug-in hybrid electric vehicle (PHEV) energy management problem, the authors demonstrate near-optimal performance, achieving a cost gap of only $4.69\%$ compared to dynamic programming (DP), while delivering sub-millisecond action execution suitable for real-time control. The approach unifies discrete gear/clutch decisions with continuous engine/motor controls, outperforming several HARL baselines and showing reasonable generalization across unseen drive cycles. While promising for real-time MIOC applications, the method carries potential constraint-violation risks, motivating future work on safety guarantees and robustness in higher-dimensional, real-world settings.

Abstract

Many optimal control problems require the simultaneous output of discrete and continuous control variables. These problems are usually formulated as mixed-integer optimal control (MIOC) problems, which are challenging to solve due to the complexity of the solution space. Numerical methods such as branch-and-bound are computationally expensive and undesirable for real-time control. This paper proposes a novel hybrid-action reinforcement learning (HARL) algorithm, twin delayed deep deterministic actor-Q (TD3AQ), for MIOC problems. TD3AQ combines the advantages of both actor-critic and Q-learning methods, and can handle the discrete and continuous action spaces simultaneously. The proposed algorithm is evaluated on a plug-in hybrid electric vehicle (PHEV) energy management problem, where real-time control of the discrete variables, clutch engagement/disengagement and gear shift, and continuous variable, engine torque, is essential to maximize fuel economy while satisfying driving constraints. Simulation outcomes demonstrate that TD3AQ achieves control results close to optimality when compared with dynamic programming (DP), with just 4.69% difference. Furthermore, it surpasses the performance of baseline reinforcement learning algorithms.

Figures (10)

• Figure 1: The forward propagation process of TD3AQ. The continuous action $a^c$ is generated by the actor network $\mu_\theta$, the subscript of $a^c$ represents the dimension of the action space, and the discrete action $a^d$ is selected by $\arg \max$ operation on the Q-values.
• Figure 2: The back propagation process of DDAQ. Assuming that the $Q_i$ has the highest Q-value, DDAQ only updates the parameters pertaining to action $a^d=i$ (the solid green lines in the figure).
• Figure 3: The parallel hybrid powertrain architecture.
• Figure 4: The engine fuel rate map, where the boundary represents the maximum engine torque.
• Figure 5: The motor efficiency map, where the upper and lower boundaries represent the maximum and minimum motor torque, respectively.