Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning

Yuxin Wu, Hongshu Guo, Ting Huang, Yue-Jiao Gong, Zeyuan Ma

TL;DR

SEEMOO addresses designer bias in surrogate management for expensive multi-objective optimization by learning a dynamic scheduling policy over a diverse surrogate pool. It combines an attention-based state extractor with a Deep Q-Network to select surrogates for NSGA-II at each step, trained across a problem distribution to maximize Pareto-front quality. Empirical results on 24 pymoo benchmarks show SEEMOO outperforms fixed-surrogate baselines and generalizes well to unseen problems, with ablations confirming the importance of the GP surrogate and IGD-based rewards. Overall, SEEMOO demonstrates the practicality of meta-RL for adaptive surrogate management in EMOOs and points to robust improvements for real-world expensive optimization tasks.

Abstract

Surrogate-assisted Evolutionary Algorithms~(SAEAs) have shown promising robustness in solving expensive optimization problems. A key aspect that impacts SAEAs' effectiveness is surrogate model selection, which in existing works is predominantly decided by human developer. Such human-made design choice introduces strong bias into SAEAs and may hurt their expected performance on out-of-scope tasks. In this paper, we propose a reinforcement learning-assisted ensemble framework, termed as SEEMOO, which is capable of scheduling different surrogate models within a single optimization process, hence boosting the overall optimization performance in a cooperative paradigm. Specifically, we focus on expensive multi-objective optimization problems, where multiple objective functions shape a compositional landscape and hence challenge surrogate selection. SEEMOO comprises following core designs: 1) A pre-collected model pool that maintains different surrogate models; 2) An attention-based state-extractor supports universal optimization state representation of problems with varied objective numbers; 3) a deep Q-network serves as dynamic surrogate selector: Given the optimization state, it selects desired surrogate model for current-step evaluation. SEEMOO is trained to maximize the overall optimization performance under a training problem distribution. Extensive benchmark results demonstrate SEEMOO's surrogate ensemble paradigm boosts the optimization performance of single-surrogate baselines. Further ablation studies underscore the importance of SEEMOO's design components.

Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning

TL;DR

SEEMOO addresses designer bias in surrogate management for expensive multi-objective optimization by learning a dynamic scheduling policy over a diverse surrogate pool. It combines an attention-based state extractor with a Deep Q-Network to select surrogates for NSGA-II at each step, trained across a problem distribution to maximize Pareto-front quality. Empirical results on 24 pymoo benchmarks show SEEMOO outperforms fixed-surrogate baselines and generalizes well to unseen problems, with ablations confirming the importance of the GP surrogate and IGD-based rewards. Overall, SEEMOO demonstrates the practicality of meta-RL for adaptive surrogate management in EMOOs and points to robust improvements for real-world expensive optimization tasks.

Abstract

Surrogate-assisted Evolutionary Algorithms~(SAEAs) have shown promising robustness in solving expensive optimization problems. A key aspect that impacts SAEAs' effectiveness is surrogate model selection, which in existing works is predominantly decided by human developer. Such human-made design choice introduces strong bias into SAEAs and may hurt their expected performance on out-of-scope tasks. In this paper, we propose a reinforcement learning-assisted ensemble framework, termed as SEEMOO, which is capable of scheduling different surrogate models within a single optimization process, hence boosting the overall optimization performance in a cooperative paradigm. Specifically, we focus on expensive multi-objective optimization problems, where multiple objective functions shape a compositional landscape and hence challenge surrogate selection. SEEMOO comprises following core designs: 1) A pre-collected model pool that maintains different surrogate models; 2) An attention-based state-extractor supports universal optimization state representation of problems with varied objective numbers; 3) a deep Q-network serves as dynamic surrogate selector: Given the optimization state, it selects desired surrogate model for current-step evaluation. SEEMOO is trained to maximize the overall optimization performance under a training problem distribution. Extensive benchmark results demonstrate SEEMOO's surrogate ensemble paradigm boosts the optimization performance of single-surrogate baselines. Further ablation studies underscore the importance of SEEMOO's design components.
Paper Structure (27 sections, 5 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 27 sections, 5 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Surrogate-Assisted Evolutionary Algorithms
  4. Meta-Black-Box Optimization
  5. Methodology
  6. Overview
  7. Low-level Optimizer
  8. Meta-level RL designs
  9. State Space
  10. Action Space
  11. Reward Function
  12. Network Design
  13. Feature Extraction Module
  14. Policy Decision Module
  15. Overall Workflow
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Overall workflow of the proposed SEEMOO framework. The optimization process is formulated as a bi-level structure, consisting of a meta-level reinforcement learning controller and a low-level evolutionary optimizer.
  • Figure 2: IGD convergence curves of SEEMOO and five baselines across various test problems.
  • Figure 3: Visual comparison of pareto fronts obtained by SEEMOO and baselines.