Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking MOEAs for solving continuous multi-objective RL problems

Carlos Hernández, Roberto Santana

TL;DR

This work addresses the problem of benchmarking MOEAs for continuous MORL by mapping MORL to a multi-objective optimization framework, proposing a feature-based MORL instance characterization, and evaluating representative MOEAs against scalarized baselines on five MuJoCo tasks. It systematically analyzes performance with HV, GD, and IGD indicators, studies PF quality, and examines how MORL configuration (e.g., episode length, network size) affects algorithm rankings. Key findings show that MOEAs exhibit strong, instance-dependent performance (e.g., NSGA2/SPEA2 often excel on 2–3 objectives, while PSO provides strong early results), and that scalarization can mask front diversity. The study provides a practical MORL benchmarking blueprint that informs MOEA design, indicator selection, and future theoretical work for robust, scalable optimization in multi-goal robotic control settings.

Abstract

Multi-objective reinforcement learning (MORL) addresses the challenge of simultaneously optimizing multiple, often conflicting, rewards, moving beyond the single-reward focus of conventional reinforcement learning (RL). This approach is essential for applications where agents must balance trade-offs between diverse goals, such as speed, energy efficiency, or stability, as a series of sequential decisions. This paper investigates the applicability and limitations of multi-objective evolutionary algorithms (MOEAs) in solving complex MORL problems. We assess whether these algorithms can effectively address the unique challenges posed by MORL and how MORL instances can serve as benchmarks to evaluate and improve MOEA performance. In particular, we propose a framework to characterize the features influencing MORL instance complexity, select representative MORL problems from the literature, and benchmark a suite of MOEAs alongside single-objective EAs using scalarized MORL formulations. Additionally, we evaluate the utility of existing multi-objective quality indicators in MORL scenarios, such as hypervolume conducting a comparison of the algorithms supported by statistical analysis. Our findings provide insights into the interplay between MORL problem characteristics and algorithmic effectiveness, highlighting opportunities for advancing both MORL research and the design of evolutionary algorithms.

Benchmarking MOEAs for solving continuous multi-objective RL problems

TL;DR

This work addresses the problem of benchmarking MOEAs for continuous MORL by mapping MORL to a multi-objective optimization framework, proposing a feature-based MORL instance characterization, and evaluating representative MOEAs against scalarized baselines on five MuJoCo tasks. It systematically analyzes performance with HV, GD, and IGD indicators, studies PF quality, and examines how MORL configuration (e.g., episode length, network size) affects algorithm rankings. Key findings show that MOEAs exhibit strong, instance-dependent performance (e.g., NSGA2/SPEA2 often excel on 2–3 objectives, while PSO provides strong early results), and that scalarization can mask front diversity. The study provides a practical MORL benchmarking blueprint that informs MOEA design, indicator selection, and future theoretical work for robust, scalable optimization in multi-goal robotic control settings.

Abstract

Multi-objective reinforcement learning (MORL) addresses the challenge of simultaneously optimizing multiple, often conflicting, rewards, moving beyond the single-reward focus of conventional reinforcement learning (RL). This approach is essential for applications where agents must balance trade-offs between diverse goals, such as speed, energy efficiency, or stability, as a series of sequential decisions. This paper investigates the applicability and limitations of multi-objective evolutionary algorithms (MOEAs) in solving complex MORL problems. We assess whether these algorithms can effectively address the unique challenges posed by MORL and how MORL instances can serve as benchmarks to evaluate and improve MOEA performance. In particular, we propose a framework to characterize the features influencing MORL instance complexity, select representative MORL problems from the literature, and benchmark a suite of MOEAs alongside single-objective EAs using scalarized MORL formulations. Additionally, we evaluate the utility of existing multi-objective quality indicators in MORL scenarios, such as hypervolume conducting a comparison of the algorithms supported by statistical analysis. Our findings provide insights into the interplay between MORL problem characteristics and algorithmic effectiveness, highlighting opportunities for advancing both MORL research and the design of evolutionary algorithms.

Paper Structure

This paper contains 30 sections, 3 equations, 38 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related work
  3. Background
  4. Related work
  5. Evolutionary reinforcement learning
  6. Multi-objective Markov decision process and MORL approaches
  7. Benchmarking of multi-objective evolutionary algorithms
  8. MORL on the multi-objective optimization framework
  9. Solving MOMDPs with MOEAs
  10. Quality indicators of MORL
  11. Characterizing MORL instances
  12. Experiments
  13. Experimental framework
  14. Test problems
  15. Performance in terms of HV
  16. ...and 15 more sections

Figures (38)

  • Figure 1: Evolution of the HV for the problem mo-hopper-v4 with $n\_episodes=5$.
  • Figure 2: Evolution of the HV for the problem mo-halfcheetah-v4 with $n\_episodes=5$.
  • Figure 3: Pareto front for the problem mo-halfcheetah-v4 with $n\_episodes=5$.
  • Figure 4: Pareto front for the problem mo-humanoid-v4 with $n\_episodes=5$.
  • Figure 5: Evolution of the GD metric for the problem mo-halfcheetah-v4 with $n\_episodes=5$.
  • ...and 33 more figures