Improving Intrinsic Exploration with Language Abstractions

TL;DR

This work investigates whether natural language abstractions can improve intrinsic exploration in reinforcement learning. By extending competitive intrinsic baselines AMIGo and NovelD to language-based variants (L-AMIGo and L-NovelD), the authors show substantial performance gains on 13 tasks in MiniGrid and MiniHack, with gains up to 85% relative. The approach provides interpretable training signals through emergent curricula and language-based novelty, and analyzes when language semantics aid exploration. The results suggest language can serve as an efficient, compositional abstraction for guiding exploration in sparse-reward domains.

Abstract

Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale to domains requiring more abstract skills. Instead, we explore natural language as a general medium for highlighting relevant abstractions in an environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 47-85% across 13 challenging tasks from the MiniGrid and MiniHack environment suites.

Figures (17)

• Figure 1: Language conveys meaningful environment abstractions. Language state annotations in the MiniGrid KeyCorridorS4R3 chevalierboisvert2018minimalistic and MiniHack Wand of Death (Hard) samvelyan2021minihack tasks.
• Figure 2: L-AMIGo. (a) AMIGo. (b) L-AMIGo teacher first predicts achievable goals, then samples a goal. (i--iii) L-AMIGo teacher training steps: updating the goal set $\mathcal{G}$ and training grounding and policy networks.
• Figure 3: Training curves. Mean extrinsic reward ($\pm$ std err) across 5 independent runs for each model and environment. In general, linguistic variants outperform their non-linguistic forms.
• Figure 4: Aggregate performance. Interquartile mean (IQM) of models across tasks. Dot is median; error bars are 95% bootstrapped CIs.
• Figure 5: One-hot performance. Models compared to variants with one-hot non-compositional goals. Plot elements same as Figure \ref{['fig:iqm']}