Language as an Abstraction for Hierarchical Deep Reinforcement Learning

TL;DR

The paper tackles learning long-horizon control with compositional abstractions by introducing HAL, a two-tier HRL framework where the high-level policy generates language instructions for a low-level, language-conditioned controller. It provides an open-source CLEVR-MuJoCo environment and a hindsight instruction relabeling technique to train the low-level policy, enabling efficient learning and generalization to new sub-skills. Empirical results show HAL outperforms state-of-the-art baselines on diverse, sparse-reward tasks and generalizes to unseen instructions and vision-based inputs. These findings underscore the value of structured language as a scalable abstraction for hierarchical planning in reinforcement learning.

Abstract

Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations. Our analysis reveals that the compositional nature of language is critical for learning diverse sub-skills and systematically generalizing to new sub-skills in comparison to non-compositional abstractions that use the same supervision.

Figures (10)

• Figure 1: The environment and some instructions that we consider in this work, along with an illustration of hindsight instruction relabeling (HIR), which we use to enable the agent to learn from many different language goals at once (Details in Section \ref{['sec:lowlevel']}).
• Figure 2: HAL: The high-level policy $\pi_{h}$ produces language instructions ${\bm{g}}$ for the low level policy $\pi_{l}$.
• Figure 3: Sample goal states for the high-level tasks in the standard (a-c) and diverse (d-f) environments. The high-level policy only receives reward if all constraints are satisfied. The global location of the objects may vary.
• Figure 4: Results for low-level policies in terms of goals accomplished per episode over training steps for HIR. Left: HIR with different number of instructions and results with non-compositional representation and with no relabeling. Middle: Results for one-hot encoded representation with increasing number of instructions. Since the one-hot cannot leverage compositionality of the language, it suffers significantly as instruction sets grow, while HIR on sentences in fact learns even faster when instruction sets increase. Right: Performance of image-based low-level policy compared against one-hot and non-compositional instruction representations.
• Figure 5: Results for high-level policy on tasks (a-c). Blue curves for HAL include the steps for training the low-level policy (a single low-level policy is used for all 3 tasks). In all settings, HAL demonstrates faster learning than DDQN. Means and standard deviations of 3 random seeds are plotted.