Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

TL;DR

The paper tackles generalization in sparse-reward, multi-task reinforcement learning by leveraging human-provided natural language instructions and action demonstrations. It proposes a hierarchical model with a high-level language generator that decomposes tasks and a low-level policy conditioned on language, trained on a large-scale crafting grid-world dataset collected from humans. Through imitation learning followed by PPO-based RL with the language generator frozen, the approach achieves strong performance on seen tasks and notably improves zero-shot generalization to unseen tasks, while also offering interpretable sub-task explanations via generated instructions. The dataset and method together demonstrate that language can serve as an effective high-level abstraction for planning and transfer in complex RL settings. This work suggests practical pathways for incorporating human language annotation into multi-task RL to enhance sample efficiency, generalization, and interpretability.

Abstract

Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting. In order to be sample efficient, multi-task learning requires reuse and sharing of low-level policies. To facilitate the automatic decomposition of hierarchical tasks, we propose the use of step-by-step human demonstrations in the form of natural language instructions and action trajectories. We introduce a dataset of such demonstrations in a crafting-based grid world. Our model consists of a high-level language generator and low-level policy, conditioned on language. We find that human demonstrations help solve the most complex tasks. We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting and to learn quickly from a few demonstrations. Generalization is not only reflected in the actions of the agent, but also in the generated natural language instructions in unseen tasks. Our approach also gives our trained agent interpretable behaviors because it is able to generate a sequence of high-level descriptions of its actions.

Figures (12)

• Figure 1: From state observation at time step $t$, the agent generates a natural language instruction "go to key and press grab," which guides the agent to grab the key. After the instruction is fulfilled and the agent grabs the key, the agent generates a new instruction at $t+1$.
• Figure 2: (Left) Example view of game interface that the worker would see on AMT. On the left the worker is given the goal and recipes; the board is in the middle; the worker provides annotations on the right. (Right) Example sequence of instructions provided by the Turker for the given task of Stone Pickaxe.
• Figure 3: (Left) High-level language generator. (Right) Low-level policy conditioned on language.
• Figure 4: Comparing baselines with our method on accuracy. Human demonstrations are necessary to complete tasks with 3 or more steps. Averaged over 3 runs.
• Figure 5: Ablation of our method with varying amounts of human annotations (25%, 50%, 75% and 100%). For each fraction, we sample that number of demonstrations from the dataset for each type of task. Averaged over 3 runs.