On the use of feature-maps and parameter control for improved quality-diversity meta-evolution

TL;DR

The paper tackles automated design of quality-diversity archives by evolving both the MAP-Elites behaviour space and its evolutionary parameters. It introduces three core advances: a k-best database to preserve quality and diversity, a general feature-map that can be non-linear and parameterised by a meta-genotype, and dynamic parameter control integrated via CMA-ES-based meta-evolution. Validated on an 8-joint planar robot arm, non-linear feature-maps yield up to ~15× higher meta-fitness than linear maps, while reinforcement learning often delivers the best parameter-control performance; the approach also enables robust damage recovery, achieving over 80% reach for moderate damages. These findings indicate that automatic configuration of QD systems can substantially improve archive quality, generalisation to unseen perturbations, and practical robustness in robotic control tasks.

Abstract

In Quality-Diversity (QD) algorithms, which evolve a behaviourally diverse archive of high-performing solutions, the behaviour space is a difficult design choice that should be tailored to the target application. In QD meta-evolution, one evolves a population of QD algorithms to optimise the behaviour space based on an archive-level objective, the meta-fitness. This paper proposes an improved meta-evolution system such that (i) the database used to rapidly populate new archives is reformulated to prevent loss of quality-diversity; (ii) the linear transformation of base-features is generalised to a feature-map, a function of the base-features parametrised by the meta-genotype; and (iii) the mutation rate of the QD algorithm and the number of generations per meta-generation are controlled dynamically. Experiments on an 8-joint planar robot arm compare feature-maps (linear, non-linear, and feature-selection), parameter control strategies (static, endogenous, reinforcement learning, and annealing), and traditional MAP-Elites variants, for a total of 49 experimental conditions. Results reveal that non-linear and feature-selection feature-maps yield a 15-fold and 3-fold improvement in meta-fitness, respectively, over linear feature-maps. Reinforcement learning ranks among top parameter control methods. Finally, our approach allows the robot arm to recover a reach of over 80% for most damages and at least 60% for severe damages.

Figures (3)

• Figure 1: Evolution of the following map quality metrics (Mean $\pm$ SD, aggregated across replicates) over the function evaluations: (a) map coverage, the number of solutions in the behaviour-performance map; (b) global fitness, the highest bottom-level fitness in the map; (b) average fitness, the mean bottom-level fitness in the map. For Meta-conditions, Mean and SD are aggregated over replicates and the behaviour-performance maps, and default hyperparameters are used (mutation rate 0.125 and 5 generations per meta-generation). Fitness is the negative variance of the angles of the robot arm when normalised to $[0,1]$ such that $0$ represents $-\pi/2rad$ and $1$ represents $-\pi/2rad$.
• Figure 2: Effect of the feature-map on meta-evolution. $x$-axis represents the number of function evaluations and $y$-axis represents the meta-fitness, the summed pairwise distance across 10% of solutions in the map averaged across the damages in which it is assessed. The average meta-fitness in the $\lambda=5$ behaviour-performance maps in the meta-population is first computed and then this average is aggregated over 5 replicates as Mean $\pm$ SD. Default hyperparameters are used (mutation rate 0.125 and 5 generations per meta-generation).
• Figure 3: Test on unseen damages that offset the joint by a particular angle. The $x$-axis represents the offset in $[-180,180]$ degrees and the $y$-axis represents the percentage of targets reached within the semi-circle span of the robot. For each offset the Mean $\pm$ SD is aggregated over 5 replicates. For Meta, the behaviour-performance map is formed from the mean meta-genotype (see $\mathbf{m}$ in Eq. \ref{['eq: multivariate normal']}) and the default hyperparameters are used (mutation rate 0.125 and 5 generations per meta-generation). Optimised indicates the best setting from parameter control (i.e., mutation rate 0.25).