Diversity-Enriched Option-Critic
Anand Kamat, Doina Precup
TL;DR
DEOC addresses degeneracy in the option-critic framework by promoting diversity among learned options via an intrinsic information-theoretic reward and a diversity-aware termination objective. The diversity bonus is defined as $R_{bonus} = \mathcal{H}(A^{\pi_{O1}}|S) + \mathcal{H}(A^{\pi_{O2}}|S) + \mathcal{H}(O^{\pi_{\Omega}}|S) + \mathcal{H}(A^{\pi_{O1}}; A^{\pi_{O2}}|S)$ and the augmented reward is $R_{aug}(S_t,A_t) = (1-\tau)R(S_t,A_t) + \tau R_{bonus}(S_t)$. The termination objective is $L(\theta_{\beta}) = \mathbb{E}[\beta(S_t,O_t) \mathcal{D}(S_t)]$, where $\mathcal{D}(S_t)$ standardizes the diversity signal from $R_{bonus}$. Empirically, DEOC and the termination variant TDEOC achieve state-of-the-art performance on discrete and continuous control benchmarks, with improved robustness, interpretability, and transferability of options compared with option-critic and PPO baselines. The results suggest that encouraging behavioral diversity in hierarchical RL improves exploration efficiency, resilience to perturbations, and reusability of learned skills in new tasks.
Abstract
Temporal abstraction allows reinforcement learning agents to represent knowledge and develop strategies over different temporal scales. The option-critic framework has been demonstrated to learn temporally extended actions, represented as options, end-to-end in a model-free setting. However, feasibility of option-critic remains limited due to two major challenges, multiple options adopting very similar behavior, or a shrinking set of task relevant options. These occurrences not only void the need for temporal abstraction, they also affect performance. In this paper, we tackle these problems by learning a diverse set of options. We introduce an information-theoretic intrinsic reward, which augments the task reward, as well as a novel termination objective, in order to encourage behavioral diversity in the option set. We show empirically that our proposed method is capable of learning options end-to-end on several discrete and continuous control tasks, outperforms option-critic by a wide margin. Furthermore, we show that our approach sustainably generates robust, reusable, reliable and interpretable options, in contrast to option-critic.