Hierarchical Decision Making by Generating and Following Natural Language Instructions

TL;DR

The paper addresses hierarchical decision making by representing high-level plans as latent natural language instructions that are generated by an instructor and executed by a separate executor. It introduces a challenging MiniRTS RTS environment and a large instruction-execution dataset, demonstrating that latent language plans with compositional encoders improve grounding and performance over direct imitation. Key contributions include a detailed multi-encoder architecture for state and instruction representation, attention over instruction history, and distinct training losses for both instruction generation and action grounding, leading to improved generalization across a large set of instructions. The work provides substantial resources, including code, models, and data, to advance language-grounded hierarchical control in complex, partially observable domains.

Abstract

We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Experiments show that models using natural language as a latent variable significantly outperform models that directly imitate human actions. The compositional structure of language proves crucial to its effectiveness for action representation. We also release our code, models and data.

Figures (7)

• Figure 1: Two agents, designated instructor and executor collaboratively play a real-time strategy game (§\ref{['section:game']}). The instructor iteratively formulates plans and issues instructions in natural language to the executor, who then executes them as a sequence of actions. We first gather a dataset of humans playing each role (§\ref{['section:dataset']}). We then train models to imitate humans actions in each role (§\ref{['section:models']}).
• Figure 2: At each time step of the environment we encode spatial observations (e.g. the game map) and non-spatial internal states for each game object (e.g. units, buildings, or resources) via the observation encoder, which produces separate feature vectors for each unit, resource, or discrete map locations. We also embed each of the last $K$ natural language instructions into individual instruction feature vectors. Lastly, we learn features for all the other global game attributes by employing the auxiliary encoder. We then use these features for both the executor and instructor networks.
• Figure 3: Modeling an action for an unit requires predicting an action type based on the global summary of current observation, and then, depending on the predicted action type, computing a probability distribution over a set of the action targets. In this case, the Move action is sampled, which uses the map cells features as the action targets.
• Figure 4: Our game implements the rock-paper-scissors attack graph, where each unit has some units it is effective against and vulnerable to.
• Figure 5: Frequency histograms for the dataset instructions and words.