CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

TL;DR

The paper introduces Contrastive Intrinsic Control (CIC), a competence-based unsupervised RL algorithm that maximizes the mutual information between state transitions and continuous latent skills using a CPC-like contrastive estimator and a particle-based entropy term. By combining state-transition entropy with a discriminative, high-dimensional skill representation, CIC fosters diverse yet predictable behaviors, enabling more efficient adaptation to downstream tasks. On the Unsupervised Reinforcement Learning Benchmark (URLB), CIC achieves leading downstream performance, surpassing prior competence-based methods by up to 79% in IQM and outperforming the next-best exploration method by 18%, demonstrating the value of large skill spaces and robust MI estimation for generalization.

Abstract

We introduce Contrastive Intrinsic Control (CIC), an algorithm for unsupervised skill discovery that maximizes the mutual information between state-transitions and latent skill vectors. CIC utilizes contrastive learning between state-transitions and skills to learn behavior embeddings and maximizes the entropy of these embeddings as an intrinsic reward to encourage behavioral diversity. We evaluate our algorithm on the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards. CIC substantially improves over prior methods in terms of adaptation efficiency, outperforming prior unsupervised skill discovery methods by 1.79x and the next leading overall exploration algorithm by 1.18x.

Figures (10)

• Figure 1: This work deals with unsupervised skill discovery through mutual information maximization. We introduce Contrastive Intrinsic Control (CIC) -- a new unsupervised RL algorithm that explores and adapts more efficiently than prior methods.
• Figure 2: Qualitative visualizations of unsupervised skills discovered in Walker, Quadruped, and Jaco arm environments. The Walker learns to balance and move, the Quadruped learns to flip upright and walk, and the 6 DOF robotic arm learns how to move without locking. Unlike prior competence-based methods for continuous control which evaluate on OpenAI Gym (e.g. EysenbachGIL19diayn), which reset the environment when the agent loses balance, CIC is able to learn skills in fixed episode length environments which are much harder to explore (see Appendix \ref{['app:gymvsdmc']}).
• Figure 3: Architecture illustrating the practical implementation of CIC . During a gradient update step, random $\tau = (s,s')$ tuples are sampled from the replay buffer, then a particle estimator is used to compute the entropy and a noise contrastive loss to compute the conditional entropy. The contrastive loss is backpropagated through the entire architecture. The entropy and contrastive terms are then scaled and added to form the intrinsic reward. The RL agent is optimized with a DDPG lillicrap15ddpg.
• Figure 4: To empirically demonstrate issues inherent to competence-based exploration methods, we run DIAYN EysenbachGIL19diayn and compare it to ICM pathak2017curiosity and a Fixed baseline where the agent receives an intrinsic reward of 1.0 for each timestep and no extrinsic reward on both OpenAI Gym (episode resets when agent loses balance) and DeepMind Control (DMC) (episode is fixed for 1k steps) Hopper environments. Since Gym and DMC rewards are on different scales, we normalize rewards based on the maximum reward achieved by any algorithm ( 1k for Gym, 3 for DMC). While DIAYN is able to achieve higher extrinsic rewards than ICM on Gym, the Fixed intrinsic reward baseline performs best. However, on DMC the Fixed and DIAYN agents achieve near-zero reward while ICM does not. This is consistent with findings of prior work that DIAYN is able to learn diverse behaviors in Gym EysenbachGIL19diayn as well as the observation that DIAYN performs poorly on DMC environments laskin_yarats_2021_urlb
• Figure 5: We report the aggregate statistics using stratified bootstrap intervals agarwal2021rliable for 12 downstream tasks on URLB with 10 seeds, so each statistic for each algorithm has 120 seeds in total. We find that overall, CIC achieves leading performance on URLB in terms of the IQM, mean, and OG statistics. As recommended by agarwal2021rliable, we use the IQM as our primary performance measure. In terms of IQM, CIC improves upon the next best skill discovery algorithm (APS) by $79\%$ and the next best algorithm overall (ProtoRL) by $18\%$.