Reinforcement Learning-assisted Constraint Relaxation for Constrained Expensive Optimization
Qianhao Zhu, Sijie Ma, Zeyuan Ma, Hongshu Guo, Yue-Jiao Gong
TL;DR
This work introduces RLECEO, a MetaBBO framework that learns a generalizable policy to adaptively control $b5$-relaxation in constrained expensive optimization by framing the optimization process as an MDP and solving it with a Double Deep Q-Network. The approach uses a constraint-aware, landscape-informed state representation and a discrete action space to balance objective improvement with constraint satisfaction, guided by a novel reward design. Empirical results on the CEC2017 CEOP suite show competitive performance against strong baselines and robust generalization across problem dimensions, with ablations confirming the importance of state, action, and reward choices as well as training. This work advances automated constraint handling by enabling learned, adaptable policies that generalize beyond the training distribution, potentially reducing reliance on hand-crafted heuristics in CEOPs.
Abstract
Constraint handling plays a key role in solving realistic complex optimization problems. Though intensively discussed in the last few decades, existing constraint handling techniques predominantly rely on human experts' designs, which more or less fall short in utility towards general cases. Motivated by recent progress in Meta-Black-Box Optimization where automated algorithm design can be learned to boost optimization performance, in this paper, we propose learning effective, adaptive and generalizable constraint handling policy through reinforcement learning. Specifically, a tailored Markov Decision Process is first formulated, where given optimization dynamics features, a deep Q-network-based policy controls the constraint relaxation level along the underlying optimization process. Such adaptive constraint handling provides flexible tradeoff between objective-oriented exploitation and feasible-region-oriented exploration, and hence leads to promising optimization performance. We train our approach on CEC 2017 Constrained Optimization benchmark with limited evaluation budget condition (expensive cases) and compare the trained constraint handling policy to strong baselines such as recent winners in CEC/GECCO competitions. Extensive experimental results show that our approach performs competitively or even surpasses the compared baselines under either Leave-one-out cross-validation or ordinary train-test split validation. Further analysis and ablation studies reveal key insights in our designs.