R2 Indicator and Deep Reinforcement Learning Enhanced Adaptive Multi-Objective Evolutionary Algorithm

TL;DR

The paper addresses the challenge of selecting effective algorithms for complex multi-objective problems by integrating a reinforcement-learning agent with a multi-objective evolutionary framework. It introduces R2-RLMOEA, which converts five single-objective EAs (GA, ES, TLBO, WOA, EO) into MOEAs using the R2 indicator and uses a Double Deep Q-learning agent to select among them each generation, guided by a reward based on the R2-based quality measure. Evaluated on the CEC09 benchmarks with IGD and SP, the approach achieves state-of-the-art performance on average and demonstrates clear patterns in EA selection across optimization stages. This work highlights the practical potential of RL to adaptively orchestrate diverse EAs for robust multi-objective optimization, paving the way for more scalable and responsive optimization systems.

Abstract

Choosing an appropriate optimization algorithm is essential to achieving success in optimization challenges. Here we present a new evolutionary algorithm structure that utilizes a reinforcement learning-based agent aimed at addressing these issues. The agent employs a double deep q-network to choose a specific evolutionary operator based on feedback it receives from the environment during optimization. The algorithm's structure contains five single-objective evolutionary algorithm operators. This single-objective structure is transformed into a multi-objective one using the R2 indicator. This indicator serves two purposes within our structure: first, it renders the algorithm multi-objective, and second, provides a means to evaluate each algorithm's performance in each generation to facilitate constructing the reinforcement learning-based reward function. The proposed R2-reinforcement learning multi-objective evolutionary algorithm (R2-RLMOEA) is compared with six other multi-objective algorithms that are based on R2 indicators. These six algorithms include the operators used in R2-RLMOEA as well as an R2 indicator-based algorithm that randomly selects operators during optimization. We benchmark performance using the CEC09 functions, with performance measured by inverted generational distance and spacing. The R2-RLMOEA algorithm outperforms all other algorithms with strong statistical significance (p<0.001) when compared with the average spacing metric across all ten benchmarks.

Figures (24)

• Figure 1: General reinforcement learning block diagram
• Figure 2: R2-RLMOEA flow chart
• Figure 3: RL reward variation over EA generations.
• Figure 4: Box plot of the IGD and SP metrics for applied algorithms on the UF1 test function.
• Figure 5: Box plot of the IGD and SP metrics for applied algorithms on the UF2 test function.