Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs
Wei Hung, Shao-Hua Sun, Ping-Chun Hsieh
TL;DR
This work tackles action-constrained RL (ACRL) with zero constraint violations, addressing the inefficiency of existing approaches that rely on quadratic programs or complex generative models. It introduces ARAM, a framework that uses Acceptance-Rejection Sampling to map unconstrained policies to feasible actions and an Augmented Unconstrained Two-Objective MDP (AUTO-MDP) to steer learning toward higher acceptance by penalizing violations, all compatible with standard RL algorithms. A Multi-Objective SAC (MOSAC) implementation enables learning policies that satisfy multiple penalty weights simultaneously via dual replay buffers and a scalarization of vector Q-values. Across MuJoCo locomotion and resource allocation benchmarks, ARAM achieves faster training, higher action-validity, and much lower QP and inference overhead, demonstrating practical, scalable benefits for safety- and resource-constrained control tasks.
Abstract
Action-constrained reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation, which is required by various safety-critical and resource-constrained applications. The existing ACRL methods can typically achieve favorable constraint satisfaction but at the cost of either high computational burden incurred by the quadratic programs (QP) or increased architectural complexity due to the use of sophisticated generative models. In this paper, we propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications: (i) To enforce the action constraints, we leverage the classic acceptance-rejection method, where we treat the unconstrained policy as the proposal distribution and derive a modified policy with feasible actions. (ii) To improve the acceptance rate of the proposal distribution, we construct an augmented two-objective Markov decision process (MDP), which include additional self-loop state transitions and a penalty signal for the rejected actions. This augmented MDP incentives the learned policy to stay close to the feasible action sets. Through extensive experiments in both robot control and resource allocation domains, we demonstrate that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the state-of-the-art ACRL methods. We have made the source code publicly available to encourage further research in this direction.