Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe Reinforcement Learning-based Control for Hydrogen Diesel Dual-Fuel Engines

Vasu Sharma, Alexander Winkler, Armin Norouzi, Jakob Andert, David Gordon, Hongsheng Guo

TL;DR

This work addresses safe, real-time control of hydrogen-diesel dual-fuel engines by integrating offline GRU-based plant modeling with offline RL agents (TD3 and PPO) using a constraint-aware reward and state augmentation. A GRU encoder-decoder DNN models engine dynamics and emissions from a dataset of 85k cycles, achieving $ ext{RMSPE} < 4.6\%$ on unseen data. The RL agents are trained in simulation and validated on a real engine, running on a Raspberry Pi with a total latency of 1 ms, which is six times faster than a comparable MPC approach. The approach is modular, data-efficient, and suitable for future online fine-tuning and application to carbon-free fuels in heavy-duty propulsion.

Abstract

The urgent energy transition requirements towards a sustainable future stretch across various industries and are a significant challenge facing humanity. Hydrogen promises a clean, carbon-free future, with the opportunity to integrate with existing solutions in the transportation sector. However, adding hydrogen to existing technologies such as diesel engines requires additional modeling effort. Reinforcement Learning (RL) enables interactive data-driven learning that eliminates the need for mathematical modeling. The algorithms, however, may not be real-time capable and need large amounts of data to work in practice. This paper presents a novel approach which uses offline model learning with RL to demonstrate safe control of a 4.5 L Hydrogen Diesel Dual-Fuel (H2DF) engine. The controllers are demonstrated to be constraint compliant and can leverage a novel state-augmentation approach for sample-efficient learning. The offline policy is subsequently experimentally validated on the real engine where the control algorithm is executed on a Raspberry Pi controller and requires 6 times less computation time compared to online Model Predictive Control (MPC) optimization.

Safe Reinforcement Learning-based Control for Hydrogen Diesel Dual-Fuel Engines

TL;DR

This work addresses safe, real-time control of hydrogen-diesel dual-fuel engines by integrating offline GRU-based plant modeling with offline RL agents (TD3 and PPO) using a constraint-aware reward and state augmentation. A GRU encoder-decoder DNN models engine dynamics and emissions from a dataset of 85k cycles, achieving on unseen data. The RL agents are trained in simulation and validated on a real engine, running on a Raspberry Pi with a total latency of 1 ms, which is six times faster than a comparable MPC approach. The approach is modular, data-efficient, and suitable for future online fine-tuning and application to carbon-free fuels in heavy-duty propulsion.

Abstract

The urgent energy transition requirements towards a sustainable future stretch across various industries and are a significant challenge facing humanity. Hydrogen promises a clean, carbon-free future, with the opportunity to integrate with existing solutions in the transportation sector. However, adding hydrogen to existing technologies such as diesel engines requires additional modeling effort. Reinforcement Learning (RL) enables interactive data-driven learning that eliminates the need for mathematical modeling. The algorithms, however, may not be real-time capable and need large amounts of data to work in practice. This paper presents a novel approach which uses offline model learning with RL to demonstrate safe control of a 4.5 L Hydrogen Diesel Dual-Fuel (H2DF) engine. The controllers are demonstrated to be constraint compliant and can leverage a novel state-augmentation approach for sample-efficient learning. The offline policy is subsequently experimentally validated on the real engine where the control algorithm is executed on a Raspberry Pi controller and requires 6 times less computation time compared to online Model Predictive Control (MPC) optimization.

Paper Structure

This paper contains 14 sections, 8 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. GRU Encoder-Decoder based DNN Engine Model
  4. Experimental Setup Details
  5. Offline Model Learning
  6. Plant Training:
  7. Results:
  8. Offline Reinforcement Learning
  9. Reward Modeling with Additive Noise
  10. State Augmentation with Perturbed Rewards
  11. RL Training
  12. RL Validation
  13. Real-Time Implementation
  14. Summary and Conclusions

Figures (7)

  • Figure 1: Modeling and controller design procedure based on deep reinforcement learning
  • Figure 2: Deep Neural Network Engine Plant Model. GRU: gated recurrent unit, DOI: duration of injection, P2M: pre to main injections, SOI: start of injection, IMEP: indicated mean effective pressure, MPRR: maximum pressure rise rate.
  • Figure 3: GRU-based network outputs on the test set
  • Figure 4: Constraint violation penalty visualized over multiple output domains.
  • Figure 5: RL training for various agents
  • ...and 2 more figures