Reinforcement Learning based Constrained Optimal Control: an Interpretable Reward Design
Jingjie Ni, Fangfei Li, Xin Jin, Xianlun Peng, Yang Tang
TL;DR
The paper tackles RL for constrained optimal control with state and terminal constraints by proposing an interpretable reward design that decomposes the objective into $\mathcal{R}(s^t,a^t)=\alpha\mathcal{R}^{a}+\beta\mathcal{R}^{g}+\lambda\mathcal{R}^{p}+\mu\mathcal{R}^{c}$ within a POMDP formulation for multi-agent systems.It provides theoretical guarantees (Theorems 1 and 2) that connect the chosen reward weights to the constrained-optimal policy, and introduces a practical two-subproblem procedure to estimate key bounds such as $\tau$ and $t_c$, enabling alignment of the RL objective with constraints.To address practical difficulties in reward calibration, the authors integrate Curriculum Learning to sequence simpler tasks before tackling the full constrained problem, ensuring convergence even with a zero guidance weight $\beta$ in final stages.Empirical results in a multi-agent particle environment show improved terminal and state constraint satisfaction and reduced control costs compared to baselines, supporting the method's potential for scalable, interpretable RL-based constrained control.
Abstract
This paper presents an interpretable reward design framework for reinforcement learning based constrained optimal control problems with state and terminal constraints. The problem is formalized within a standard partially observable Markov decision process framework. The reward function is constructed from four weighted components: a terminal constraint reward, a guidance reward, a penalty for state constraint violations, and a cost reduction incentive reward. A theoretically justified reward design is then presented, which establishes bounds on the weights of the components. This approach ensures that constraints are satisfied and objectives are optimized while mitigating numerical instability. Acknowledging the importance of prior knowledge in reward design, we sequentially solve two subproblems, using each solution to inform the reward design for the subsequent problem. Subsequently, we integrate reinforcement learning with curriculum learning, utilizing policies derived from simpler subproblems to assist in tackling more complex challenges, thereby facilitating convergence. The framework is evaluated against original and randomly weighted reward designs in a multi-agent particle environment. Experimental results demonstrate that the proposed approach significantly enhances satisfaction of terminal and state constraints and optimization of control cost.