Multi-Objective Covariance Matrix Adaptation MAP-Annealing

TL;DR

MO-CMA-MAE introduces a CMA-ES-based MOQD algorithm that optimizes Hypervolume Improvement within thresholded cell fronts to jointly promote exploration of under-explored measure-space cells and improvement of discovered Pareto Sets. By coupling CMA-ES with a threshold-annealing mechanism, the method maintains a dynamic trade-off between exploring new cells and refining existing ones, achieving superior MOQD-scores and broader archive coverage in several domains, including Overcooked map generation. The approach outperforms baselines such as MOME, NSGA-II, SMS-EMOA, and COMO-CMA-ES on multiple tasks and demonstrates strong performance even in 3-objective variants, while highlighting the computational cost of exact hypervolume calculations as a limitation and a direction for future work.

Abstract

Quality-Diversity (QD) optimization is an emerging field that focuses on finding a set of behaviorally diverse and high-quality solutions. While the quality is typically defined w.r.t. a single objective function, recent work on Multi-Objective Quality-Diversity (MOQD) extends QD optimization to simultaneously optimize multiple objective functions. This opens up multi-objective applications for QD, such as generating a diverse set of game maps that maximize difficulty, realism, or other properties. Existing MOQD algorithms use non-adaptive methods such as mutation and crossover to search for non-dominated solutions and construct an archive of Pareto Sets (PS). However, recent work in QD has demonstrated enhanced performance through the use of covariance-based evolution strategies for adaptive solution search. We propose bringing this insight into the MOQD problem, and introduce MO-CMA-MAE, a new MOQD algorithm that leverages Covariance Matrix Adaptation-Evolution Strategies (CMA-ES) to optimize the hypervolume associated with every PS within the archive. We test MO-CMA-MAE on three MOQD domains, and for generating maps of a co-operative video game, showing significant improvements in performance.

Figures (13)

• Figure 1: Game maps found by MO-CMA-MAE that induce diverse task divisions while achieving good trade-off between the difficulty and emptiness objectives. In this case, the two task divisions we consider are the difference between numbers of ingredients held by the two cooperating players, and difference between numbers of plates held. In the two upper-left maps, the blue player starts between the plate (represented by three white circles) and the delivery counter (represented by a gray tile), and thus is induced to specialize in delivering cooked meals on plates to the counter. The two upper-right maps induce the opposite, where the green player specializes in delivery.
• Figure 2: The gap between $T_e$ and $F_e$ boosts HVI for a new solution, and should be narrower at well-explored trade-offs.
• Figure 3: When inserting a new solution with objectives ${\bf f}({\bf x}_i)$ to its assigned threshold front $T_e$, MO-CMA-MAE searches for a discount factor $d_i$ satisfying $\hbox{HVI}_r(d_i{\bf f}({\bf x}_i), T_e) \approx \alpha\hbox{HVI}_r({\bf f}({\bf x}_i), T_e)$.
• Figure 4: MOQD-scores and Coverages achieved by tested algorithms on each domain.
• Figure 5: Heatmaps representing the passive MOQD archive populated by each algorithm after 5000 iterations of optimization on the sphere domain. The color intensity within each cell corresponds to the hypervolume within it.