Pre-trained Language Models as Prior Knowledge for Playing Text-based Games

TL;DR

This paper tackles the sample-inefficiency of RL in text-based games by injecting pre-trained language priors into RL agents. By fine-tuning DistilBERT on game transcripts and using it to encode state and actions for DRRN/TDQN, the approach supplies linguistic and world-knowledge priors that guide policy learning without per-game LM fine-tuning. Empirical results on the Jericho suite show state-of-the-art scores on several games (including Zork1) and strong generalization signals, with ablations confirming the value of pretraining. The work highlights the potential of transformer priors to enhance language understanding and decision-making in goal-directed, partially observable textual environments, and suggests avenues for broader generalization and commonsense integration.

Abstract

Recently, text world games have been proposed to enable artificial agents to understand and reason about real-world scenarios. These text-based games are challenging for artificial agents, as it requires an understanding of and interaction using natural language in a partially observable environment. Agents observe the environment via textual descriptions designed to be challenging enough for even human players. Past approaches have not paid enough attention to the language understanding capability of the proposed agents. Typically, these approaches train from scratch, an agent that learns both textual representations and the gameplay online during training using a temporal loss function. Given the sample-inefficiency of RL approaches, it is inefficient to learn rich enough textual representations to be able to understand and reason using the textual observation in such a complicated game environment setting. In this paper, we improve the semantic understanding of the agent by proposing a simple RL with LM framework where we use transformer-based language models with Deep RL models. We perform a detailed study of our framework to demonstrate how our model outperforms all existing agents on the popular game, Zork1, to achieve a score of 44.7, which is 1.6 higher than the state-of-the-art model. Overall, our proposed approach outperforms 4 games out of the 14 text-based games, while performing comparable to the state-of-the-art models on the remaining games.

Figures (7)

• Figure 1: A sample gameplay by our model (DBERT-DRRN) for the classic text game, Zork1. The aim is to solve puzzles and collect 19 treasures in the trophy case, but the agent is not aware of the goal and learns from rewards. Each white box is a state of the game at a step; the green and orange boxes are the action taken, and the reward received correspondingly. The state contains location if visible (underlined), current observation (black), inventory with the player (red), and the current location description (blue). In this example, location, inventory, and description are only mentioned when it changes, while the complete state information is provided to the agent during gameplay. This example shows how our model is able to perform better by learning to use the egg and the lantern in the correct way, as highlighted.
• Figure 2: RL agents - (a) DRRN: Q-value Q(o,a) is computed for observation o and action a, (b) TDQN: Q-values Q(o,u), Q(o,$p_1$), Q(o,$p_2$) for all templates $u \in T$ and all vocabulary $p \in V$.
• Figure 3: DBERT performance ablation results on Zork1
• Figure 4: DBERT-DRRN performance results on Inhumane
• Figure 5: DBERT-DRRN performance results on Jewel