Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals

Octavio Pappalardo

TL;DR

This work addresses the challenge of producing transferable reinforcement learning policies without relying on extrinsic rewards during pre-training. It introduces ULEE, an unsupervised meta-learning framework that learns an in-context policy while simultaneously generating self-imposed goals through an adversarial curriculum guided by a post-adaptation difficulty metric. By coupling Goal Proposal, Goal Selection, and Difficulty Prediction, ULEE maintains a curriculum at intermediate difficulty and optimizes exploration and rapid adaptation across diverse tasks. Empirical results on XLand-MiniGrid demonstrate superior zero-shot and few-shot performance, improved fine-tuning initializations, and robust generalization to new goals, dynamics, and map structures, outperforming DIAYN, PPO, RND, and RL$^{2}$ baselines. The work provides a scalable pathway to foundation policies and suggests future directions in hierarchical architectures and multimodal goal specification.

Abstract

Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks. A promising direction, grounded in human development, investigates agents that learn by setting and pursuing their own goals. The core challenge lies in how to effectively generate, select, and learn from such goals. Our focus is on broad distributions of downstream tasks where solving every task zero-shot is infeasible. Such settings naturally arise when the target tasks lie outside of the pre-training distribution or when their identities are unknown to the agent. In this work, we (i) optimize for efficient multi-episode exploration and adaptation within a meta-learning framework, and (ii) guide the training curriculum with evolving estimates of the agent's post-adaptation performance. We present ULEE, an unsupervised meta-learning method that combines an in-context learner with an adversarial goal-generation strategy that maintains training at the frontier of the agent's capabilities. On XLand-MiniGrid benchmarks, ULEE pre-training yields improved exploration and adaptation abilities that generalize to novel objectives, environment dynamics, and map structures. The resulting policy attains improved zero-shot and few-shot performance, and provides a strong initialization for longer fine-tuning processes. It outperforms learning from scratch, DIAYN pre-training, and alternative curricula.

Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals

TL;DR

This work addresses the challenge of producing transferable reinforcement learning policies without relying on extrinsic rewards during pre-training. It introduces ULEE, an unsupervised meta-learning framework that learns an in-context policy while simultaneously generating self-imposed goals through an adversarial curriculum guided by a post-adaptation difficulty metric. By coupling Goal Proposal, Goal Selection, and Difficulty Prediction, ULEE maintains a curriculum at intermediate difficulty and optimizes exploration and rapid adaptation across diverse tasks. Empirical results on XLand-MiniGrid demonstrate superior zero-shot and few-shot performance, improved fine-tuning initializations, and robust generalization to new goals, dynamics, and map structures, outperforming DIAYN, PPO, RND, and RL baselines. The work provides a scalable pathway to foundation policies and suggests future directions in hierarchical architectures and multimodal goal specification.

Abstract

Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks. A promising direction, grounded in human development, investigates agents that learn by setting and pursuing their own goals. The core challenge lies in how to effectively generate, select, and learn from such goals. Our focus is on broad distributions of downstream tasks where solving every task zero-shot is infeasible. Such settings naturally arise when the target tasks lie outside of the pre-training distribution or when their identities are unknown to the agent. In this work, we (i) optimize for efficient multi-episode exploration and adaptation within a meta-learning framework, and (ii) guide the training curriculum with evolving estimates of the agent's post-adaptation performance. We present ULEE, an unsupervised meta-learning method that combines an in-context learner with an adversarial goal-generation strategy that maintains training at the frontier of the agent's capabilities. On XLand-MiniGrid benchmarks, ULEE pre-training yields improved exploration and adaptation abilities that generalize to novel objectives, environment dynamics, and map structures. The resulting policy attains improved zero-shot and few-shot performance, and provides a strong initialization for longer fine-tuning processes. It outperforms learning from scratch, DIAYN pre-training, and alternative curricula.
Paper Structure (28 sections, 5 equations, 5 figures, 2 tables)

This paper contains 28 sections, 5 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Pre-trained Policy
  5. Goal Curriculum
  6. Goal Proposal
  7. Goal Selection
  8. Difficulty Prediction
  9. Experiments
  10. Benchmarks and Tasks
  11. Baselines and Ablations
  12. Experimental Results
  13. Exploration
  14. Fast Adaptation
  15. Fine-tuning on Fixed Tasks
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Panel (a), reproduced from nikulin2024xland (https://creativecommons.org/licenses/by/4.0/), shows a goal and the rules that must be triggered to achieve it, represented as a tree of depth 2. Analogously, tasks from the trivial and small benchmarks correspond to depth-0 and depth-1 trees. Panels (b)-(d) show example environments from the three benchmarks: 4Rooms-Trivial, 4Rooms-Small, and 6Rooms-Small.
  • Figure 2: Percentage of $\mu^{\text{eval}}$ goals reached under different exploration budgets. A goal is considered reached at episode $j$ if it was achieved in any episode $\leq j$. Results are averaged across 4 seeds, with individual seeds overlaid as faint thin lines.
  • Figure 3: Evaluations on $\mu^{\text{eval}}$ tasks: (a) performance across episodes during task-specific adaptation, (b) few-shot performance by task percentile, (c) few-shot performance as pre-training on $\mu^{\text{unsup}}$ progresses. ULEE pre-training improves over baselines and ablations across all views. The legend in (c) applies to all panels. Reported steps for ULEE variants in (c) omit those from the Goal-search Policy, which adds 25%. Results are averaged over 4 seeds, with shaded regions indicating standard deviation.
  • Figure 4: Mean, 40th, and 20th percentile returns on a fixed set of $\mu^{\text{eval}}$ tasks as learning on them progresses. Results are averaged over 4 seeds, with shaded regions indicating standard deviation.
  • Figure 5: Mean, 40th, and 20th percentile returns on $\mu^{\text{eval}}$ tasks as meta-learning on $\mu^{\text{train}}$ progresses. Results are averaged over 4 seeds, with shaded regions indicating standard deviation.