Extract-QD Framework: A Generic Approach for Quality-Diversity in Noisy, Stochastic or Uncertain Domains

TL;DR

The paper addresses the challenge of applying Quality-Diversity in uncertain domains by introducing the Extract-QD Framework (EQD), a modular toolbox that unifies existing Uncertain-QD (UQD) approaches, and Extract-ME (EME), a robust first-guess method for such tasks. EME employs adaptive-sampling through an extraction mechanism, performance estimation buffers, and an archive depth to re-evaluate elites within a fixed evaluation budget, achieving strong performance across standard UQD benchmarks. The EQD Framework demonstrates its utility by enabling task-specific adaptations, exemplified by Extract-PGA (EPGA), which augments PGA-ME with uncertainty handling and yields substantial gains without increasing evaluation cost. Empirical results show EME consistently matching or surpassing the best existing methods and that integrating EQD insights into PGA-ME significantly enhances performance, jointly lowering the barrier to deploying UQD in real-world QD applications. These contributions offer a practical pathway to harness uncertainty-aware QD in diverse domains, from robotics to content generation, with scalable, modular design.

Abstract

Quality-Diversity (QD) has demonstrated potential in discovering collections of diverse solutions to optimisation problems. Originally designed for deterministic environments, QD has been extended to noisy, stochastic, or uncertain domains through various Uncertain-QD (UQD) methods. However, the large number of UQD methods, each with unique constraints, makes selecting the most suitable one challenging. To remedy this situation, we present two contributions: first, the Extract-QD Framework (EQD Framework), and second, Extract-ME (EME), a new method derived from it. The EQD Framework unifies existing approaches within a modular view, and facilitates developing novel methods by interchanging modules. We use it to derive EME, a novel method that consistently outperforms or matches the best existing methods on standard benchmarks, while previous methods show varying performance. In a second experiment, we show how our EQD Framework can be used to augment existing QD algorithms and in particular the well-established Policy-Gradient-Assisted-MAP-Elites method, and demonstrate improved performance in uncertain domains at no additional evaluation cost. For any new uncertain task, our contributions now provide EME as a reliable "first guess" method, and the EQD Framework as a tool for developing task-specific approaches. Together, these contributions aim to lower the cost of adopting UQD insights in QD applications.

Figures (10)

• Figure 1: Comparison of the (A) standard QD setup, where each solution gets a single deterministic fitness and descriptor feature value, and the (B) UQD setup, where each solution gets a distribution of these values. We propose the Extract-QD Framework and Extract-MAP-Elites for UQD, equivalent respectively to the QD-Framework and MAP-Elites for QD.
• Figure 2: Comparison of (A) the existing ME algorithm with (B) our proposed EME algorithm, which is designed to address the uncertainty present in UQD tasks. EME introduces an extraction mechanism that re-evaluates solutions within the archive, an estimation mechanism and a depth to the archive.
• Figure 3: Illustration of (A) the existing QD Framework for the QD setup, and (B) our proposed EQD Framework that extends it for uncertain tasks. The EQD Framework encompasses existing UQD approaches and allows practitioners to easily build new tailored approaches for UQD tasks.
• Figure 4: Comparison of our EME approach with similar existing UQD approaches. The Corrected QD-Score reflects performance on the task, while the Average Samples offer insights into how the methods manage the available sampling budget. The vertical line represents the median across $10$ replications; the box shows the quartiles, the whiskers indicate $1.5$ times the interquartile range, and the dots represent outliers. Blank lines indicate undefined approach-task pairs.
• Figure 5: Final Corrected Archive of a random replication of each approach across all tasks. Blank panels indicate undefined approach-task pairs. Each cell corresponds to a solution, with brighter colours indicating higher fitness. The axes represent the descriptor dimensions detailed in Table \ref{['tab:tasks']}.