Soft Quality-Diversity Optimization

TL;DR

The paper reframes Quality-Diversity (QD) by introducing Soft QD, a discretization-free objective that uses a Gaussian illumination model over the continuous behavior space. It derives SQUAD, a differentiable algorithm that optimizes a lower bound of the Soft QD Score, balancing high quality with spread in behavior space through pairwise repulsion terms. The approach is analyzed theoretically (monotonicity, submodularity, and limiting connections to traditional QD) and validated empirically across high-dimensional benchmarks, showing strong scalability and competitive performance against state-of-the-art baselines. It also demonstrates a practical mechanism to control the quality-diversity trade-off via a kernel bandwidth and discusses efficiency techniques (batches, nearest neighbors) and bounded-space handling via logit transforms. Overall, Soft QD broadens the applicability of QD to differentiable, high-dimensional domains and provides a scalable, tuneable framework for producing diverse, high-quality solution sets.

Abstract

Quality-Diversity (QD) algorithms constitute a branch of optimization that is concerned with discovering a diverse and high-quality set of solutions to an optimization problem. Current QD methods commonly maintain diversity by dividing the behavior space into discrete regions, ensuring that solutions are distributed across different parts of the space. The QD problem is then solved by searching for the best solution in each region. This approach to QD optimization poses challenges in large solution spaces, where storing many solutions is impractical, and in high-dimensional behavior spaces, where discretization becomes ineffective due to the curse of dimensionality. We present an alternative framing of the QD problem, called \emph{Soft QD}, that sidesteps the need for discretizations. We validate this formulation by demonstrating its desirable properties, such as monotonicity, and by relating its limiting behavior to the widely used QD Score metric. Furthermore, we leverage it to derive a novel differentiable QD algorithm, \emph{Soft QD Using Approximated Diversity (SQUAD)}, and demonstrate empirically that it is competitive with current state of the art methods on standard benchmarks while offering better scalability to higher dimensional problems.

Figures (10)

• Figure 1: Left: In a discrete archive, each cell is fully illuminated by its highest-quality occupant. Right: In Soft QD, each solution illuminates the area around with an intensity proportional to its quality. The smooth scalar field defined by the behavior value $v_{\boldsymbol{\theta}}(\mathbf{b})$ is independent of discretization.
• Figure 2: Optimizing Soft QD Score with SQUAD. The plots visualize the behavior value function, $v_{\boldsymbol{\theta}}(\mathbf{b})$, induced by a population of five solutions. The bottom plane represents the behavior space $\mathcal{B}$, the height corresponding to solution quality, $f$, and the colored surface shows the induced behavior value $v_{\boldsymbol{\theta}}$. Initially (left), a cluster of low-quality solutions induces a low behavior value. As SQUAD improves the population (right), the induced behavior value increases in both magnitude and coverage, leading to a higher Soft QD Score.
• Figure 3: QVS (left) and QD Score (right) on LP tasks with increasing behavior descriptor dimensionality ($4$, $8$, $16$). All results are averaged over $10$ runs, with error bars depicting the standard errors. SQUAD's performance relative to the baselines improves with task complexity, with it outperforming all other methods on the most challenging $16$-d task for both metrics.
• Figure 4: Controlling the quality-diversity tradeoff with $\gamma^2$. The plot shows how varying $\gamma^2$ impacts solution quality, measured by the mean objective (blue line), and diversity, measured by the Vendi Score (dashed red line).
• Figure 5:

Theorems & Definitions (16)

• Theorem 1: Informally stated
• Theorem 2
• Theorem 2
• proof
• Lemma 1: Bonferroni Inequalities for the Maximum
• proof
• Lemma 2: Bounding the Truncation Error
• proof
• Theorem 3: Monotonicity with respect to population size
• proof