Multi-Objective Reinforcement Learning for Energy-Efficient Industrial Control

TL;DR

The paper addresses energy-efficient industrial control by applying Multi-Objective Reinforcement Learning (MORL) to balance tracking accuracy and power consumption on the Quanser Aero 2 1-DoF pitch system. It introduces a composite reward $R_t = - (1 - \alpha) \cdot |\Delta_t| - \alpha \cdot P_t$ with a tunable $\alpha$ to navigate the trade-off, and conducts extensive simulations and real-world tests across multiple $\alpha$ values, observing a sharp performance shift between $\alpha = 0.0$ and $\alpha = 0.25$. The results indicate that low $\alpha$ drives aggressive, non-Pareto behavior potentially due to optimizer artifacts from Adam, while higher $\alpha$ yields smoother, Pareto-optimal performance and better sim-to-real transfer; the authors propose GP-based Pareto-front modeling and MOBO to automate $\alpha$ selection and improve sample efficiency. Overall, the work advances energy-aware control in industrial CPS and outlines a path toward real deployment on the Quanser Aero 2, with future work focusing on robust Pareto modeling and optimization-driven deployment strategies.

Abstract

Industrial automation increasingly demands energy-efficient control strategies to balance performance with environmental and cost constraints. In this work, we present a multi-objective reinforcement learning (MORL) framework for energy-efficient control of the Quanser Aero 2 testbed in its one-degree-of-freedom configuration. We design a composite reward function that simultaneously penalizes tracking error and electrical power consumption. Preliminary experiments explore the influence of varying the Energy penalty weight, alpha, on the trade-off between pitch tracking and energy savings. Our results reveal a marked performance shift for alpha values between 0.0 and 0.25, with non-Pareto optimal solutions emerging at lower alpha values, on both the simulation and the real system. We hypothesize that these effects may be attributed to artifacts introduced by the adaptive behavior of the Adam optimizer, which could bias the learning process and favor bang-bang control strategies. Future work will focus on automating alpha selection through Gaussian Process-based Pareto front modeling and transitioning the approach from simulation to real-world deployment.

Figures (2)

• Figure 1: Block diagram of the system model.
• Figure 2: Pareto front of RL solutions for energy-efficient control.