diff History for Neural Language Agents

TL;DR

This work addresses data efficiency in neural language agents for sequential decision-making by introducing diff history, a delta-based prompt abstraction that replaces verbose full-text observation histories with salient changes computed via Unix diff. The authors formalize how to compute and use diff histories both during finetuning and inference, and demonstrate substantial gains on two diverse environments: BabyAI-Text and NetHack (LangHack). In NetHack, diff history enables state-of-the-art performance with 1800x fewer labeled demonstrations, while in BabyAI-Text it yields a significant reduction in training steps required to reach comparable performance; token-level analysis shows diff acts as a soft compression in the high-dimensional NetHack setting. The results suggest that task-agnostic abstractions in prompts can dramatically improve data efficiency and generalization for LM-based agents, with open-source data and code provided for broader use and further development.

Abstract

Neural Language Models (LMs) offer an exciting solution for general-purpose embodied control. However, a key technical issue arises when using an LM-based controller: environment observations must be converted to text, which coupled with history, results in long and verbose textual prompts. As a result, prior work in LM agents is limited to restricted domains with small observation size as well as minimal needs for interaction history or instruction tuning. In this paper, we introduce diff history, a simple and highly effective solution to these issues. By applying the Unix diff command on consecutive text observations in the interaction histories used to prompt LM policies, we can both abstract away redundant information and focus the content of textual inputs on the salient changes in the environment. On NetHack, an unsolved video game that requires long-horizon reasoning for decision-making, LMs tuned with diff history match state-of-the-art performance for neural agents while needing 1800x fewer training examples compared to prior work. Even on the simpler BabyAI-Text environment with concise text observations, we find that although diff history increases the length of prompts, the representation it provides offers a 25% improvement in the efficiency of low-sample instruction tuning. Further, we show that diff history scales favorably across different tuning dataset sizes. We open-source our code and data to https://diffhistory.github.io.

Figures (10)

• Figure 1: Instruction finetuning LMs for sequential decision-making with diff history: We summarize our key results on the unsolved video-game NetHack (left) and illustrate our proposed approach (right). Neural language agents with diff history match state-of-the-art performance on NetHack while training on orders of magnitude fewer data. diff history improves the quality of generations in low-resource decision-making domains by replacing full-text observations in interaction history data provided in-context to LMs as "memory" with richly structured deltas. These deltas are computed using an off-the-shelf implementation of the Unix diff operation.
• Figure 2: A closer look at the text deltas produced by diff: we provide examples of full-text and diff observations in our two environments of interest, BabyAI-Text and NetHack. A more detailed example of Unix-style diff outputs can be found in Appendix \ref{['appendix:more_on_diff']}.
• Figure 3: Mean token counts per observation, before and after diff: We compute token statistics of full-text and diff observations across natural language interaction datasets generated for a low-dimensional environment (BabyAI-Text) and a high-dimensional environment (NetHack) via the tokenizer we use for all LM experiments radford2019language. Error bars indicate one standard deviation across datasets.
• Figure 4: Low-resource BabyAI-Text: We compare the performance of language agents tuned with diff history against full-text history ablations in two low-sample finetuning settings, with 1,000 demonstrations (left) and 5,000 demonstrations (right) respectively. Though both agents eventually achieve equivalent performance over the course of finetuning in both experiments, the diff history language agent requires $\geq$25% fewer FLOPs to attain this peak. Error bars indicate one standard error in mean reward on a batch of withheld task instances.
• Figure 5: Low-resource NetHack: We compare the performance of language agents with diff history against ablations, testing efficiency and generalization when LMs are tuned on demonstrations of interaction history for natural language action prediction (left) and joint action-world model prediction (right). Error bars indicated one standard error in mean game score achieved by agents across a batch of withheld NetHack instances.