Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics

Luca Grillotti, Maxence Faldor, Borja G. León, Antoine Cully

TL;DR

Quality-Diversity Actor-Critic (QDAC) presents a dual-critic framework that unifies value-based quality and successor-features-based diversity through a constrained optimization with an adaptive Lagrange multiplier. By conditioning a policy on explicit skills, QDAC learns a broad set of high-performing behaviors and demonstrates improved diversity and performance across six continuous-control locomotion tasks, along with robust few-shot adaptation and hierarchical learning capabilities. The paper provides both a model-free SAC-based and a model-based DreamerV3-based variant, with extensive comparisons to Quality-Diversity baselines from evolutionary and RL perspectives, and ablations confirming the importance of successor features and adaptive constraint balancing. These results highlight the practical potential of combining value and behavior representations to create versatile, adaptable agents, and point to future work in unsupervised skill discovery and non-ergodic environments.

Abstract

A key aspect of intelligence is the ability to demonstrate a broad spectrum of behaviors for adapting to unexpected situations. Over the past decade, advancements in deep reinforcement learning have led to groundbreaking achievements to solve complex continuous control tasks. However, most approaches return only one solution specialized for a specific problem. We introduce Quality-Diversity Actor-Critic (QDAC), an off-policy actor-critic deep reinforcement learning algorithm that leverages a value function critic and a successor features critic to learn high-performing and diverse behaviors. In this framework, the actor optimizes an objective that seamlessly unifies both critics using constrained optimization to (1) maximize return, while (2) executing diverse skills. Compared with other Quality-Diversity methods, QDAC achieves significantly higher performance and more diverse behaviors on six challenging continuous control locomotion tasks. We also demonstrate that we can harness the learned skills to adapt better than other baselines to five perturbed environments. Finally, qualitative analyses showcase a range of remarkable behaviors: adaptive-intelligent-robotics.github.io/QDAC.

Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics

TL;DR

Quality-Diversity Actor-Critic (QDAC) presents a dual-critic framework that unifies value-based quality and successor-features-based diversity through a constrained optimization with an adaptive Lagrange multiplier. By conditioning a policy on explicit skills, QDAC learns a broad set of high-performing behaviors and demonstrates improved diversity and performance across six continuous-control locomotion tasks, along with robust few-shot adaptation and hierarchical learning capabilities. The paper provides both a model-free SAC-based and a model-based DreamerV3-based variant, with extensive comparisons to Quality-Diversity baselines from evolutionary and RL perspectives, and ablations confirming the importance of successor features and adaptive constraint balancing. These results highlight the practical potential of combining value and behavior representations to create versatile, adaptable agents, and point to future work in unsupervised skill discovery and non-ergodic environments.

Abstract

A key aspect of intelligence is the ability to demonstrate a broad spectrum of behaviors for adapting to unexpected situations. Over the past decade, advancements in deep reinforcement learning have led to groundbreaking achievements to solve complex continuous control tasks. However, most approaches return only one solution specialized for a specific problem. We introduce Quality-Diversity Actor-Critic (QDAC), an off-policy actor-critic deep reinforcement learning algorithm that leverages a value function critic and a successor features critic to learn high-performing and diverse behaviors. In this framework, the actor optimizes an objective that seamlessly unifies both critics using constrained optimization to (1) maximize return, while (2) executing diverse skills. Compared with other Quality-Diversity methods, QDAC achieves significantly higher performance and more diverse behaviors on six challenging continuous control locomotion tasks. We also demonstrate that we can harness the learned skills to adapt better than other baselines to five perturbed environments. Finally, qualitative analyses showcase a range of remarkable behaviors: adaptive-intelligent-robotics.github.io/QDAC.
Paper Structure (49 sections, 3 theorems, 39 equations, 33 figures, 13 tables)

This paper contains 49 sections, 3 theorems, 39 equations, 33 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Statement
  4. Methods
  5. Actor Objective
  6. Practical Algorithm
  7. Experiments
  8. Tasks
  9. Learning Diverse High-Performing Skills
  10. Harnessing Skills for Few-Shot Adaptation and Hierarchical Learning
  11. Baselines
  12. Quality-Diversity via Evolutionary Algorithms
  13. Quality-Diversity via Reinforcement Learning
  14. Ablations
  15. Evaluation Metrics
  16. ...and 34 more sections

Key Result

Proposition 1

Consider an infinite horizon, finite MDP with observable features in $\Phi$. Let $\pi$ be a policy and let ${\boldsymbol{\mathbf{\psi}}}$ be the discounted successor features. Then, for all skills ${\boldsymbol{\mathbf{z}}} \in \mathcal{Z}$, we can derive an upper bound for the distance between ${\b

Figures (33)

  • Figure 1: a) QDAC's architecture: the agent $\pi(a | s, {\boldsymbol{\mathbf{z}}})$ learns high-performing and diverse behaviors with a dual critics optimization $V(s, {\boldsymbol{\mathbf{z}}})$ and ${\boldsymbol{\mathbf{\psi}}}(s, {\boldsymbol{\mathbf{z}}})$ which are balanced with a Lagrange multiplier $\lambda(s, {\boldsymbol{\mathbf{z}}})$. b) Example of diverse behaviors on a set of challenging continuous control tasks. c) Few-shot adaptation tasks and hierarchical learning tasks using the diversity of skills learned by QDAC.
  • Figure 2: The Lagrange multiplier is optimized to balance the quality-diversity trade-off, see Eq. \ref{['eq:lagrange-obj']}. a) If the expected features $(1 - \gamma) {\boldsymbol{\mathbf{\psi}}}(s, {\boldsymbol{\mathbf{z}}})$ is in the neighborhood of ${\boldsymbol{\mathbf{z}}}$, then $\lambda(s, {\boldsymbol{\mathbf{z}}})$ decreases to focus on maximizing the return. b) Otherwise, $\lambda(s, {\boldsymbol{\mathbf{z}}})$ increases to focus on executing ${\boldsymbol{\mathbf{z}}}$. c) After the Lagrange multiplier is updated, the policy is optimized according to the objective.
  • Figure 3: Distance and performance scores normalized and aggregated across all tasks. The values correspond to the IQM while the error bars represent IQM 95% CI.
  • Figure 4: (top) Distance profiles and (bottom) performance profiles for each task defined in Section \ref{['sec:experiments-metrics']}. The lines represent the IQM for 10 replications, and the shaded areas correspond to the 95% CI. Figure \ref{['fig:profiles_explanation']} illustrates how to read distance and performance profiles.
  • Figure 5: Ant Velocity Heatmaps of (top) negative distance to skill, (bottom) performance defined in Section \ref{['sec:experiments-metrics']}. The heatmap represents the skill space $\mathcal{Z} = [-5 \text{ m/s}, 5 \text{ m/s}]^2$, of target velocities. This space is discretized into cells, with each cell representing a distinct skill ${\boldsymbol{\mathbf{z}}} = v_xv_y^\intercal$. In the bottom row, empty cells show which skills are not successfully executed (i.e. $d({\boldsymbol{\mathbf{z}}}) > d_{\mathrm{eval}}$). The heatmaps for other tasks are presented in section \ref{['appendix:heatmaps']}.
  • ...and 28 more figures

Theorems & Definitions (5)

  • Proposition
  • Proposition
  • proof
  • Proposition 2.1
  • proof