Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agents

Vincenzo De Paola, Mirco Mutti, Riccardo Zamboni, Marcello Restelli

TL;DR

Reinforcement learning is notoriously sample-inefficient, even with parallel rollout capabilities. K-Myriad proposes unsupervised pretraining in a parallel MDP setting by maximizing the collective state entropy $H(d_{ pi_p})$ across a population of policies, implemented as a shared trunk with multiple heads and trained via a $k$-NN entropy estimator. The approach scales to hundreds or thousands of replicas and yields a diverse portfolio of specialized exploration strategies that can jump-start downstream RL by selecting the most task-aligned head for initialization. Empirical results on high-dimensional locomotion tasks in Isaac Sim show that increased policy diversity leads to richer behaviors and faster early learning, illustrating a scalable path to exploiting parallelism for exploration and pretraining in RL.

Abstract

Parallelization in Reinforcement Learning is typically employed to speed up the training of a single policy, where multiple workers collect experience from an identical sampling distribution. This common design limits the potential of parallelization by neglecting the advantages of diverse exploration strategies. We propose K-Myriad, a scalable and unsupervised method that maximizes the collective state entropy induced by a population of parallel policies. By cultivating a portfolio of specialized exploration strategies, K-Myriad provides a robust initialization for Reinforcement Learning, leading to both higher training efficiency and the discovery of heterogeneous solutions. Experiments on high-dimensional continuous control tasks, with large-scale parallelization, demonstrate that K-Myriad can learn a broad set of distinct policies, highlighting its effectiveness for collective exploration and paving the way towards novel parallelization strategies.

K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agents

TL;DR

Reinforcement learning is notoriously sample-inefficient, even with parallel rollout capabilities. K-Myriad proposes unsupervised pretraining in a parallel MDP setting by maximizing the collective state entropy across a population of policies, implemented as a shared trunk with multiple heads and trained via a -NN entropy estimator. The approach scales to hundreds or thousands of replicas and yields a diverse portfolio of specialized exploration strategies that can jump-start downstream RL by selecting the most task-aligned head for initialization. Empirical results on high-dimensional locomotion tasks in Isaac Sim show that increased policy diversity leads to richer behaviors and faster early learning, illustrating a scalable path to exploiting parallelism for exploration and pretraining in RL.

Abstract

Parallelization in Reinforcement Learning is typically employed to speed up the training of a single policy, where multiple workers collect experience from an identical sampling distribution. This common design limits the potential of parallelization by neglecting the advantages of diverse exploration strategies. We propose K-Myriad, a scalable and unsupervised method that maximizes the collective state entropy induced by a population of parallel policies. By cultivating a portfolio of specialized exploration strategies, K-Myriad provides a robust initialization for Reinforcement Learning, leading to both higher training efficiency and the discovery of heterogeneous solutions. Experiments on high-dimensional continuous control tasks, with large-scale parallelization, demonstrate that K-Myriad can learn a broad set of distinct policies, highlighting its effectiveness for collective exploration and paving the way towards novel parallelization strategies.
Paper Structure (40 sections, 9 equations, 13 figures, 3 tables)

This paper contains 40 sections, 9 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Unsupervised Exploration in Parallel MDPs
  5. The Parallel State Entropy Objective
  6. Scalable Parallel Algorithms for the State Entropy Objective
  7. Policy Architecture
  8. Loss Function
  9. The Algorithm
  10. Jump Starting RL with a Pretrained Parallel Policy
  11. Illustrative Example
  12. Experiments
  13. Unsupervised Parallel Exploration
  14. Pretrained Locomotion
  15. Conclusions
  16. ...and 25 more sections

Figures (13)

  • Figure 1: Visualization of the k-NN entropy estimator (Eq. \ref{['eq:maxentropyestimator']}).
  • Figure 2: The parallel agent-environment interaction.
  • Figure 3: K-Myriad policy architecture. A shared backbone extracts features from the input, while independent heads produce distinct policy outputs.
  • Figure 4: The PointMaze environment. (a) Pretraining with K-Myriad and resulting heatmaps of two parallel agents. (b,c) Jump-starting RL for different goal states with the two pretrained policies (heatmaps after training). Goal cells are in green, and the initial state is in red.
  • Figure 5: Parallel state entropy achieved by a single generalist agent, against a collection of 10 or 50 agents. The plots report the average state entropy and 95% c.i. across 4 runs as a function of the policy updates.
  • ...and 8 more figures