Quality with Just Enough Diversity in Evolutionary Policy Search

TL;DR

This work tackles the limitations of Evolution Strategies in deceptive fitness landscapes by introducing Quality with Just Enough Diversity (JEDi), which leverages behavior descriptors to guide a Gaussian Process (GP) that maps behavior to fitness. JEDi selects promising target behaviors via a Weighted GP and uses parallel ES to optimize policies near those targets, updating the GP with evaluated solutions to refine future targets. A Weighted Gaussian Process accounts for the exploration budget per behavior cell, and target selection is performed through a Pareto front over GP mean and uncertainty, with a Weighted Target Fitness Score $ SWTFS^{\alpha}$ balancing proximity to targets and environment fitness: $\nSWTFS^{\alpha}(\phi) = \alpha \nS_{target}(\phi) + (1-\alpha) \nS_{fitness}(\phi)$, where $\nS_{target}(\phi) = 1 - \frac{\| d(\phi) - d_{target} \| - d_{min}}{d_{max} - d_{min}}$ and $\nS_{fitness}(\phi) = \frac{f(\phi) - f_{min}}{f_{max} - f_{min}}$. Empirically, JEDi outperforms both QD and ES baselines on hard maze exploration and Brax control tasks, demonstrating improved sample efficiency and the ability to uncover high-fitness policies in deceptive landscapes. The approach offers a principled path to harness behavior information for robust policy optimization and suggests fruitful future hybrids with CMA-MAE and PGA-ME components.

Abstract

Evolution Strategies (ES) are effective gradient-free optimization methods that can be competitive with gradient-based approaches for policy search. ES only rely on the total episodic scores of solutions in their population, from which they estimate fitness gradients for their update with no access to true gradient information. However this makes them sensitive to deceptive fitness landscapes, and they tend to only explore one way to solve a problem. Quality-Diversity methods such as MAP-Elites introduced additional information with behavior descriptors (BD) to return a population of diverse solutions, which helps exploration but leads to a large part of the evaluation budget not being focused on finding the best performing solution. Here we show that behavior information can also be leveraged to find the best policy by identifying promising search areas which can then be efficiently explored with ES. We introduce the framework of Quality with Just Enough Diversity (JEDi) which learns the relationship between behavior and fitness to focus evaluations on solutions that matter. When trying to reach higher fitness values, JEDi outperforms both QD and ES methods on hard exploration tasks like mazes and on complex control problems with large policies.

Figures (3)

• Figure 1: Fitness and number of solutions tried in each behavior cell for MAP-Elites (ME), Evolution Strategies (ES), and JEDi on Walker2D.
• Figure 2: Mazes for robot exploration. Robots start at the blue point and the target is the green circle.
• Figure 3: Convergence and final fitness distribution on the quadrupled maze problem.