Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Reinforcement Learning-Assisted Automated Operator Portfolio for Constrained Multi-objective Optimization

Shuai Shao, Ye Tian, Shangshang Yang, Xingyi Zhang

Abstract

Constrained multi-objective optimization problems (CMOPs) are of great significance in the context of practical applications, ranging from scientific to engineering domains. Most existing constrained multi-objective evolutionary algorithms (CMOEAs) usually employ fixed operators all the time, which exhibit poor versatility in handling various CMOPs. Therefore, some recent studies have focused on adaptively selecting the best operators for the current population states during the search process. The evolutionary algorithms proposed in these studies learn the value of each operator and recommend the operator with the highest value for the current population, resulting in only a single operator being recommended at each generation, which can potentially lead to local optima and inefficient utilization of function evaluations. To address the dilemma in operator adaptation, this paper proposes a reinforcement learning-based automated operator portfolio approach to learn an allocation scheme of operators at each generation. This approach considers the optimization-related and constraint-related features of the current population as states, the overall improvement in population convergence and diversity as rewards, and different operator portfolios as actions. By utilizing deep neural networks to establish a mapping model between the population states and the expected cumulative rewards, the proposed approach determines the optimal operator portfolio during the evolutionary process. By embedding the proposed approach into existing CMOEAs, a deep reinforcement learning-assisted automated operator portfolio based evolutionary algorithm for solving CMOPs, abbreviated as CMOEA-AOP, is developed. Empirical studies on 33 benchmark problems demonstrate that the proposed algorithm significantly enhances the performance of CMOEAs and exhibits more stable performance across different CMOPs.

Deep Reinforcement Learning-Assisted Automated Operator Portfolio for Constrained Multi-objective Optimization

Abstract

Constrained multi-objective optimization problems (CMOPs) are of great significance in the context of practical applications, ranging from scientific to engineering domains. Most existing constrained multi-objective evolutionary algorithms (CMOEAs) usually employ fixed operators all the time, which exhibit poor versatility in handling various CMOPs. Therefore, some recent studies have focused on adaptively selecting the best operators for the current population states during the search process. The evolutionary algorithms proposed in these studies learn the value of each operator and recommend the operator with the highest value for the current population, resulting in only a single operator being recommended at each generation, which can potentially lead to local optima and inefficient utilization of function evaluations. To address the dilemma in operator adaptation, this paper proposes a reinforcement learning-based automated operator portfolio approach to learn an allocation scheme of operators at each generation. This approach considers the optimization-related and constraint-related features of the current population as states, the overall improvement in population convergence and diversity as rewards, and different operator portfolios as actions. By utilizing deep neural networks to establish a mapping model between the population states and the expected cumulative rewards, the proposed approach determines the optimal operator portfolio during the evolutionary process. By embedding the proposed approach into existing CMOEAs, a deep reinforcement learning-assisted automated operator portfolio based evolutionary algorithm for solving CMOPs, abbreviated as CMOEA-AOP, is developed. Empirical studies on 33 benchmark problems demonstrate that the proposed algorithm significantly enhances the performance of CMOEAs and exhibits more stable performance across different CMOPs.
Paper Structure (15 sections, 9 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 15 sections, 9 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work and Motivation
  3. Constrained Multi-objective Optimization
  4. Existing Operator Adaptation Approaches
  5. Motivation of This Work
  6. The Proposed Algorithm
  7. Framework of CMOEA-AOP
  8. Automated Operation Portfolio
  9. Agent Training
  10. Empirical Studies
  11. Settings of Problems
  12. Settings of Algorithms
  13. Experimental Results
  14. Ablation Studies
  15. Conclusion

Figures (5)

  • Figure 1: Convergence profiles of IGD values obtained by EMCMO1 (using only genetic operators), EMCMO2 (using only differential evolution operators), EMCMO3 (using both genetic and differential evolution operators) on CF2, CF6, and CF9.
  • Figure 2: Illustration of the proposed CMOEA-AOP with deep reinforcement learning based automated operator portfolio, where a DDPG agent consisting of an actor network and a critic network is employed to recommend the optimal operator portfolio scheme for CMOEAs. Here "States" represent the optimization-related and constraint-related features of the current population, and "Action" represents the optimal operator portfolio scheme recommended by the deep reinforcement learning agent based on the current evolutionary states.
  • Figure 3: Populations with the median IGD obtained by EMCMO, Bico, AGEMOEA-II, DRLOS, and the proposed CMOEA-AOP on LIR-CMOP12.
  • Figure 4: Populations with the median IGD obtained by EMCMO, Bico, AGEMOEA-II, DRLOS, and the proposed CMOEA-AOP on DAS-CMOP8.
  • Figure 5: Convergence profiles of mean IGD values by EMCMO, Bico, AGEMOEA-II, DRLOS, and the proposed CMOEA-AOP on CF6, LIR-CMOP3 and LIR-CMOP4.