Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Constrained Multi-objective Optimization with Deep Reinforcement Learning Assisted Operator Selection

Fei Ming, Wenyin Gong, Ling Wang, Yaochu Jin

TL;DR

The paper addresses the challenge of selecting appropriate evolutionary operators for constrained multi-objective optimization by introducing a DRL-assisted online operator selection framework. A Deep Q-Network learns to map population-state features (convergence, diversity, feasibility) to operator choices, guided by a reward reflecting improvements in population quality, and is embedded into four CMOEAs. Empirical results on four CMOP suites across 42 problems show enhanced performance and versatility against nine state-of-the-art CMOEAs, with robustness to parameter settings. The approach offers a scalable, constraint-aware mechanism to adapt operators during evolution, enhancing convergence and Pareto coverage in practice.

Abstract

Solving constrained multi-objective optimization problems with evolutionary algorithms has attracted considerable attention. Various constrained multi-objective optimization evolutionary algorithms (CMOEAs) have been developed with the use of different algorithmic strategies, evolutionary operators, and constraint-handling techniques. The performance of CMOEAs may be heavily dependent on the operators used, however, it is usually difficult to select suitable operators for the problem at hand. Hence, improving operator selection is promising and necessary for CMOEAs. This work proposes an online operator selection framework assisted by Deep Reinforcement Learning. The dynamics of the population, including convergence, diversity, and feasibility, are regarded as the state; the candidate operators are considered as actions; and the improvement of the population state is treated as the reward. By using a Q-Network to learn a policy to estimate the Q-values of all actions, the proposed approach can adaptively select an operator that maximizes the improvement of the population according to the current state and thereby improve the algorithmic performance. The framework is embedded into four popular CMOEAs and assessed on 42 benchmark problems. The experimental results reveal that the proposed Deep Reinforcement Learning-assisted operator selection significantly improves the performance of these CMOEAs and the resulting algorithm obtains better versatility compared to nine state-of-the-art CMOEAs.

Constrained Multi-objective Optimization with Deep Reinforcement Learning Assisted Operator Selection

TL;DR

The paper addresses the challenge of selecting appropriate evolutionary operators for constrained multi-objective optimization by introducing a DRL-assisted online operator selection framework. A Deep Q-Network learns to map population-state features (convergence, diversity, feasibility) to operator choices, guided by a reward reflecting improvements in population quality, and is embedded into four CMOEAs. Empirical results on four CMOP suites across 42 problems show enhanced performance and versatility against nine state-of-the-art CMOEAs, with robustness to parameter settings. The approach offers a scalable, constraint-aware mechanism to adapt operators during evolution, enhancing convergence and Pareto coverage in practice.

Abstract

Solving constrained multi-objective optimization problems with evolutionary algorithms has attracted considerable attention. Various constrained multi-objective optimization evolutionary algorithms (CMOEAs) have been developed with the use of different algorithmic strategies, evolutionary operators, and constraint-handling techniques. The performance of CMOEAs may be heavily dependent on the operators used, however, it is usually difficult to select suitable operators for the problem at hand. Hence, improving operator selection is promising and necessary for CMOEAs. This work proposes an online operator selection framework assisted by Deep Reinforcement Learning. The dynamics of the population, including convergence, diversity, and feasibility, are regarded as the state; the candidate operators are considered as actions; and the improvement of the population state is treated as the reward. By using a Q-Network to learn a policy to estimate the Q-values of all actions, the proposed approach can adaptively select an operator that maximizes the improvement of the population according to the current state and thereby improve the algorithmic performance. The framework is embedded into four popular CMOEAs and assessed on 42 benchmark problems. The experimental results reveal that the proposed Deep Reinforcement Learning-assisted operator selection significantly improves the performance of these CMOEAs and the resulting algorithm obtains better versatility compared to nine state-of-the-art CMOEAs.
Paper Structure (27 sections, 13 equations, 5 figures, 5 tables, 3 algorithms)

This paper contains 27 sections, 13 equations, 5 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work and Motivations
  3. Preliminaries
  4. Adaptive Operator Selection in MOPs
  5. DRL and its Applications in MOPs
  6. Applications in Real-world MOPs
  7. Applications in MOEAs
  8. Motivations
  9. Proposed Methods
  10. Proposed DRL Model
  11. Proposed DQL-assisted Framework
  12. Train/Update the DQN
  13. Proposed OS Method
  14. Computational Complexity
  15. Remarks
  16. ...and 12 more sections

Figures (5)

  • Figure 1: An illustration of two types of working principles of the DQL technique.
  • Figure 2: The illustration of the proposed DQL model.
  • Figure 3: The flowchart of the proposed DQL-assisted CMOEA framework.
  • Figure 4: The final solution sets obtained by DRLOS-EMCMO and other methods on DAS-CMOP9 with the median IGD+ value among $30$ runs.
  • Figure 5: The convergence profiles on IGD+ of DRLOS-EMCMO and other methods on CF4, DAS-CMOP1, DOC7, and LIR-CMOP6 with the median IGD+ values among $30$ runs.