Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies

TL;DR

This work proposes an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model, which exploits a three-level hierarchy of both discrete and continuous latent variables, to capture a set of high-level behaviours while allowing for variance in how they are executed.

Abstract

For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model. In contrast to existing work, our method exploits a three-level hierarchy of both discrete and continuous latent variables, to capture a set of high-level behaviours while allowing for variance in how they are executed. We demonstrate in manipulation domains that the method can effectively cluster offline data into distinct, executable behaviours, while retaining the flexibility of a continuous latent variable model. The resulting skills can be transferred and fine-tuned on new tasks, unseen objects, and from state to vision-based policies, yielding better sample efficiency and asymptotic performance compared to existing skill- and imitation-based methods. We further analyse how and when the skills are most beneficial: they encourage directed exploration to cover large regions of the state space relevant to the task, making them most effective in challenging sparse-reward settings.

Figures (11)

• Figure 1: (a) Graphical model for HeLMS, with solid lines indicating the underlying generative model (prior) and dashed lines indicating dependencies introduced by the inference model (posterior). (b) Network architecture, showing the high-, mid-, and low-level networks from left to right, respectively. As indicated by superscripts, different subsets of the input state $\mathbf{x}$ can be provided to the high level ($HL$), mid level ($ML$), and low level ($LL$) (information-asymmetry).
• Figure 2: The object sets (triplets) used for our experiments, introduced by 2021BeyondPT.
• Figure 3: (a) Image sequences showing example rollouts when fixing the discrete skill (different for each row) and running the mid- and low-level controllers in the environment. Each skill executes a different behaviour, such as lifting (top row), reach-to-red (middle row), or grasping (bottom) (b) The learned skill transition prior $p(\mathbf{y}_t \, | \, \mathbf{y}_{t-1})$. (c) Histogram showing the use of different skills.
• Figure 4: Performance when transferring to the red-on-blue stacking task using staged sparse reward with every unseen object set.
• Figure 5: (a) Performance on pyramid task; and (b) image sequence showing episode rollout from a learned solution on this task (left-to-right, top-to-bottom).