Reinforcement learning Based Automated Design of Differential Evolution Algorithm for Black-box Optimization
Xu Yang, Rui Wang, Kaiwen Li, Ling Wang
TL;DR
This work tackles the challenge of selecting and configuring a Differential Evolution (DE) algorithm for black-box optimization without manual tuning. It introduces rlDE, an RL-based meta-optimizer that uses Exploratory Landscape Analysis (ELA) to extract problem characteristics and a MADQN agent to map these features to a tailored DE design (initialization, mutation, crossover, and control parameters) implemented within a μ+λ DE framework. Trained offline on a broad set of BBOPs and evaluated on the BBOB2009 suite, rlDE demonstrated superior aggregated performance (AEI) and competitive, stable results across multiple test functions, outperforming several state-of-the-art RL-assisted DE methods. The approach addresses the No Free Lunch challenge by prioritizing problem characteristics to generate problem-specific DEs before evolution, potentially reducing manual design effort and improving adaptability to unseen problems. Future work will expand the design space, refine reward structures, and explore additional RL architectures to further improve robustness and generalization.
Abstract
Differential evolution (DE) algorithm is recognized as one of the most effective evolutionary algorithms, demonstrating remarkable efficacy in black-box optimization due to its derivative-free nature. Numerous enhancements to the fundamental DE have been proposed, incorporating innovative mutation strategies and sophisticated parameter tuning techniques to improve performance. However, no single variant has proven universally superior across all problems. To address this challenge, we introduce a novel framework that employs reinforcement learning (RL) to automatically design DE for black-box optimization through meta-learning. RL acts as an advanced meta-optimizer, generating a customized DE configuration that includes an optimal initialization strategy, update rule, and hyperparameters tailored to a specific black-box optimization problem. This process is informed by a detailed analysis of the problem characteristics. In this proof-of-concept study, we utilize a double deep Q-network for implementation, considering a subset of 40 possible strategy combinations and parameter optimizations simultaneously. The framework's performance is evaluated against black-box optimization benchmarks and compared with state-of-the-art algorithms. The experimental results highlight the promising potential of our proposed framework.