Guided Meta-Policy Search

TL;DR

Guided Meta-Policy Search (GMPS) tackles meta-reinforcement learning’s sample inefficiency by decoupling per-task expert learning from meta-learning, and by using supervised imitation to guide the outer optimization. The method allows leveraging off-policy expert policies or demonstrations, along with a DAgger-style data aggregation scheme, to stabilize and accelerate meta-training, even in vision-based and sparse-reward tasks. Empirical results show significant improvements in meta-training sample efficiency over prior methods (MAML, PEARL) and strong performance when learning from demonstrations, including in high-dimensional visual domains. This approach broadens the practical applicability of meta-RL to real-world robotics and complex control tasks by enabling faster adaptation with fewer environment interactions during meta-training.

Abstract

Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples since they learn from scratch. Meta-RL aims to address this challenge by leveraging experience from previous tasks so as to more quickly solve new tasks. However, in practice, these algorithms generally also require large amounts of on-policy experience during the meta-training process, making them impractical for use in many problems. To this end, we propose to learn a reinforcement learning procedure in a federated way, where individual off-policy learners can solve the individual meta-training tasks, and then consolidate these solutions into a single meta-learner. Since the central meta-learner learns by imitating the solutions to the individual tasks, it can accommodate either the standard meta-RL problem setting or a hybrid setting where some or all tasks are provided with example demonstrations. The former results in an approach that can leverage policies learned for previous tasks without significant amounts of on-policy data during meta-training, whereas the latter is particularly useful in cases where demonstrations are easy for a person to provide. Across a number of continuous control meta-RL problems, we demonstrate significant improvements in meta-RL sample efficiency in comparison to prior work as well as the ability to scale to domains with visual observations.

Figures (7)

• Figure 1: Overview of the guided meta-policy search algorithm: We learn a policy $\pi_{\theta}$ which is capable of fast adaptation to new tasks via reinforcement learning, by using reinforcement learning in the inner loop of optimization and supervised learning in the meta-optimization. This algorithm either trains per-task experts $\pi_i^*$ or assumes that they are provided by human demonstrations, and then uses this for meta-optimization. Importantly, when faced with a new task we can simply perform standard reinforcement learning via policy gradient, and the policy will quickly adapt to new tasks because of the meta-training.
• Figure 2: Illustration of pushing (left), door opening (center) and legged locomotion (right) used in our experiments, with the goal regions specified in green for pushing and locomotion.
• Figure 3: Meta-training efficiency on full state pushing and dense reward locomotion. All methods reach similar asymptotic performance, but GMPS requires significantly fewer samples.
• Figure 4: Test-time extrapolation for dense reward ant locomotion. The test tasks involve navigating to the red goals indicated (right). GMPS gets better average return across tasks (left).
• Figure 5: Meta-training comparisons for sparse reward door opening (left), sparse reward ant locomotion (middle) and vision pusher (right). Our method is able to learn when only sparse rewards are available for adaptation, whereas prior methods struggle. For vision-based tasks, we find that GMPS is able to effectively leverage the demonstrations to quickly and stably learn to adapt.

Theorems & Definitions (2)

• Theorem 4.1
• Theorem A.1