Density Descent for Diversity Optimization

TL;DR

This paper addresses diversity optimization by introducing Density Descent Search (DDS), which guides CMA-ES with continuous density estimates of the feature space to preferentially explore low-density regions. DDS presents two density-estimation variants, KDE and CNF, and proves that NS is a special case of DDS-KDE under certain conditions while also establishing that KDE offers stronger stability than novelty-based measures. Empirical results across multiple DO/QD benchmarks show DDS-KDE and DDS-CNF achieving higher coverage and better exploration (lower cross-entropy) than state-of-the-art baselines, especially in higher-dimensional feature spaces. The work highlights the benefits of continuous, stable density representations for adaptive optimizers and outlines future extensions to other density models and complex domains.

Abstract

Diversity optimization seeks to discover a set of solutions that elicit diverse features. Prior work has proposed Novelty Search (NS), which, given a current set of solutions, seeks to expand the set by finding points in areas of low density in the feature space. However, to estimate density, NS relies on a heuristic that considers the k-nearest neighbors of the search point in the feature space, which yields a weaker stability guarantee. We propose Density Descent Search (DDS), an algorithm that explores the feature space via CMA-ES on a continuous density estimate of the feature space that also provides a stronger stability guarantee. We experiment with DDS and two density estimation methods: kernel density estimation (KDE) and continuous normalizing flow (CNF). On several standard diversity optimization benchmarks, DDS outperforms NS, the recently proposed MAP-Annealing algorithm, and other state-of-the-art baselines. Additionally, we prove that DDS with KDE provides stronger stability guarantees than NS, making it more suitable for adaptive optimizers. Furthermore, we prove that NS is a special case of DDS that descends a KDE of the feature space.

Figures (5)

• Figure 1: We propose density descent search (DDS) for solving diversity optimization (DO) problems. DDS first draws solutions from a Gaussian $\mathcal{N}({\bm{\mu}}, {\bm{\Sigma}})$. After computing the solution features (in this case, the final position of the robot in a maze), DDS ranks solutions by density. This density ranking is passed to CMA-ES, which updates the search distribution to sample solutions with lower density on the next iteration. Concurrently, solutions are stored in a buffer that forms the basis for density estimates, and in a passive archive that tracks all discovered solutions.
• Figure 2:
• Figure 3: Coverage and cross-entropy (CE) after 5,000 iterations of each algorithm in all domains. We report the mean over 10 trials, with error bars showing the standard error of the mean. Higher coverage and lower cross-entropy are better.
• Figure 4: Coverage and cross-entropy (CE) after 5,000 iterations of DDS-KDE in all domains for each normalized bandwidth $h_0$. We report the mean over 10 trials (3 trials for Deceptive Maze) with error bars showing the standard error of the mean. Higher coverage and lower cross-entropy are better. Highlighted cells are results from the main experiments in \ref{['fig:results']}. The plots show the normalized bandwidth on the $x$-axis
• Figure 5: Heatmaps of DDS-KDE, DDS-CNF, NS, and CMA-MAE in the three domains that have 2D feature spaces (LP, Arm Repertoire, Deceptive Maze).

Theorems & Definitions (13)

• Definition 5.1
• Theorem 5.2
• Theorem 5.3
• Theorem 5.4
• Conjecture 5.5
• Theorem C.1
• proof
• Theorem E.1
• proof
• Theorem E.2