Hybrid Reinforcement Learning and Model Predictive Control for Adaptive Control of Hydrogen-Diesel Dual-Fuel Combustion

TL;DR

This work tackles robust control of hydrogen–diesel dual-fuel engines under plant drift by proposing a hybrid RL-ML-MPC framework. An RL agent dynamically offsets the ML-MPC load reference to track changing conditions, while the ML-MPC ensures safe, constrained optimization of engine inputs. Experimental results on a single-cylinder Cummins QSB engine show ML-MPC alone yields an IMEP RMSE of 0.57 bar under injector aging, which improves to 0.44 bar with RL adaptation, albeit with an RL-induced offset and occasional overshoot bounded by safety constraints. The approach demonstrates a viable path for drift-tolerant, real-time adaptive control in dual-fuel engines and points to future work in granting RL broader control over MPC parameters.

Abstract

Reinforcement Learning (RL) and Machine Learning Integrated Model Predictive Control (ML-MPC) are promising approaches for optimizing hydrogen-diesel dual-fuel engine control, as they can effectively control multiple-input multiple-output systems and nonlinear processes. ML-MPC is advantageous for providing safe and optimal controls, ensuring the engine operates within predefined safety limits. In contrast, RL is distinguished by its adaptability to changing conditions through its learning-based approach. However, the practical implementation of either method alone poses challenges. RL requires high variance in control inputs during early learning phases, which can pose risks to the system by potentially executing unsafe actions, leading to mechanical damage. Conversely, ML-MPC relies on an accurate system model to generate optimal control inputs and has limited adaptability to system drifts, such as injector aging, which naturally occur in engine applications. To address these limitations, this study proposes a hybrid RL and ML-MPC approach that uses an ML-MPC framework while incorporating an RL agent to dynamically adjust the ML-MPC load tracking reference in response to changes in the environment. At the same time, the ML-MPC ensures that actions stay safe throughout the RL agent's exploration. To evaluate the effectiveness of this approach, fuel pressure is deliberately varied to introduce a model-plant mismatch between the ML-MPC and the engine test bench. The result of this mismatch is a root mean square error (RMSE) in indicated mean effective pressure of 0.57 bar when running the ML-MPC. The experimental results demonstrate that RL successfully adapts to changing boundary conditions by altering the tracking reference while ML-MPC ensures safe control inputs. The quantitative improvement in load tracking by implementing RL is an RSME of 0.44 bar.

Figures (2)

• Figure 1: Hybrid Reinforcement Learning and Model Predictive Control Approach.
• Figure 2: Comparison of Control Results of MPC and Hybrid RL/ML-MPC approach with model-plant mismatch.