Quality Diversity for Robot Learning: Limitations and Future Directions

TL;DR

This work critiques traditional MAP-Elites-style QD approaches for imposing bounded archives and relying on many independent policies, proposing instead a unified framework where a single goal-conditioned policy is paired with a graph-based planner to explore and generalize in maze-like tasks. By treating the archive as a relational graph and using Dijkstra's algorithm for planning, the method achieves state-of-the-art coverage and descriptor accuracy on standard QD benchmarks with significantly reduced memory footprint and training time. The authors connect QD to cognitive maps from neuroscience, arguing that learning task-invariant structure and factorized latent representations can enable truly open-ended search. They outline future directions involving neural memory modules and world models to bridge QD with cognitive maps and enhance generalization across task variants.

Abstract

Quality Diversity (QD) has shown great success in discovering high-performing, diverse policies for robot skill learning. While current benchmarks have led to the development of powerful QD methods, we argue that new paradigms must be developed to facilitate open-ended search and generalizability. In particular, many methods focus on learning diverse agents that each move to a different xy position in MAP-Elites-style bounded archives. Here, we show that such tasks can be accomplished with a single, goal-conditioned policy paired with a classical planner, achieving O(1) space complexity w.r.t. the number of policies and generalization to task variants. We hypothesize that this approach is successful because it extracts task-invariant structural knowledge by modeling a relational graph between adjacent cells in the archive. We motivate this view with emerging evidence from computational neuroscience and explore connections between QD and models of cognitive maps in human and other animal brains. We conclude with a discussion exploring the relationships between QD and cognitive maps, and propose future research directions inspired by cognitive maps towards future generalizable algorithms capable of truly open-ended search.

Figures (1)

• Figure 1: Corrected QD metrics: QD-Score, Coverage (Cov), Best Reward (Best), and Descriptor Error Mean (DEM). Results are averaged over 5 seeds for trap and 3 seeds for maze (due to computational cost arising from the episode length).