Detect and Act: Automated Dynamic Optimizer through Meta-Black-Box Optimization
Zijian Gao, Yuanting Zhong, Zeyuan Ma, Yue-Jiao Gong, Hongshu Guo
TL;DR
Meta-DO introduces an end-to-end reinforcement learning framework for automated variation detection and self-adaptation in dynamic optimization problems, replacing hand-crafted detect-then-act pipelines with a bi-level Meta Black-Box Optimization approach. A deep Q-network-based meta-policy guides a NBNC-PSO low-level optimizer, using continuous actions to finely adjust inertia and acceleration parameters while leveraging an elite archive and a rich state representation to sense landscape drift. Trained on a distribution of non-stationary problems with a PPO-based objective, Meta-DO generalizes to unseen DOPs and demonstrates superior performance on 32 benchmark instances and a real-time USV path-planning task, highlighting its cross-domain applicability and robustness under tight computational budgets. The results show Meta-DO consistently outperforms state-of-the-art baselines, validating the value of end-to-end dynamic adaptation in evolutionary computation and offering practical benefits for real-time, non-stationary optimization tasks.
Abstract
Dynamic Optimization Problems (DOPs) are challenging to address due to their complex nature, i.e., dynamic environment variation. Evolutionary Computation methods are generally advantaged in solving DOPs since they resemble dynamic biological evolution. However, existing evolutionary dynamic optimization methods rely heavily on human-crafted adaptive strategy to detect environment variation in DOPs, and then adapt the searching strategy accordingly. These hand-crafted strategies may perform ineffectively at out-of-box scenarios. In this paper, we propose a reinforcement learning-assisted approach to enable automated variation detection and self-adaption in evolutionary algorithms. This is achieved by borrowing the bi-level learning-to-optimize idea from recent Meta-Black-Box Optimization works. We use a deep Q-network as optimization dynamics detector and searching strategy adapter: It is fed as input with current-step optimization state and then dictates desired control parameters to underlying evolutionary algorithms for next-step optimization. The learning objective is to maximize the expected performance gain across a problem distribution. Once trained, our approach could generalize toward unseen DOPs with automated environment variation detection and self-adaption. To facilitate comprehensive validation, we further construct an easy-to-difficult DOPs testbed with diverse synthetic instances. Extensive benchmark results demonstrate flexible searching behavior and superior performance of our approach in solving DOPs, compared to state-of-the-art baselines.