Transfer of Reinforcement Learning-Based Controllers from Model- to Hardware-in-the-Loop

TL;DR

This paper tackles the data- and safety-driven challenges of applying reinforcement learning to embedded powertrain control by proposing a transfer-learning and X-in-the-Loop pipeline that trains in Model-in-the-Loop (MiL) and fine-tunes in Hardware-in-the-Loop (HiL). The RL objective is framed as $J(\pi_{\theta})=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)]$ over an environment modeled as a discrete-time MDP $(S,A,P,R)$ with policy $\pi_{\theta}$. Results show that TL reduces HiL training time by up to a factor of 5.9 and yields emissions improvements, illustrating a strong synergy between TL and XiL for rapid, safe deployment of RL controllers in automotive applications. The work provides a practical framework for data-efficient, real-world RL deployment in powertrain control, achieving better performance than the baseline ECU with substantially less hardware training effort.

Abstract

The process of developing control functions for embedded systems is resource-, time-, and data-intensive, often resulting in sub-optimal cost and solutions approaches. Reinforcement Learning (RL) has great potential for autonomously training agents to perform complex control tasks with minimal human intervention. Due to costly data generation and safety constraints, however, its application is mostly limited to purely simulated domains. To use RL effectively in embedded system function development, the generated agents must be able to handle real-world applications. In this context, this work focuses on accelerating the training process of RL agents by combining Transfer Learning (TL) and X-in-the-Loop (XiL) simulation. For the use case of transient exhaust gas re-circulation control for an internal combustion engine, use of a computationally cheap Model-in-the-Loop (MiL) simulation is made to select a suitable algorithm, fine-tune hyperparameters, and finally train candidate agents for the transfer. These pre-trained RL agents are then fine-tuned in a Hardware-in-the-Loop (HiL) system via TL. The transfer revealed the need for adjusting the reward parameters when advancing to real hardware. Further, the comparison between a purely HiL-trained and a transferred agent showed a reduction of training time by a factor of 5.9. The results emphasize the necessity to train RL agents with real hardware, and demonstrate that the maturity of the transferred policies affects both training time and performance, highlighting the strong synergies between TL and XiL simulation.

Figures (6)

• Figure 1: Schematics of HiL system.
• Figure 2: Framework architecture on MiL (left) and HiL (right), with training process based on TL.
• Figure 3: Progress, repetitions and transferred agents (A-D) in MiL phase during training with PPO.
• Figure 4: Impact of NOX reward factor variation based on WLTC after transfer on the HiL from MiL after 150.0 cycles (agent C).
• Figure 5: Full MiL, TL, and full HiL training based on the WLTC.