Multi-Stage Episodic Control for Strategic Exploration in Text Games

TL;DR

The paper tackles the hard exploration-exploitation trade-off in text-based adventure games, where action spaces are vast and rewards sparse. It proposes eXploit-Then-eXplore (XTX), a two-stage episodic control method that first exploits by imitating promising past trajectories and then explores with an inverse-dynamics–augmented policy, combined via a tunable mixture. Across 12 Jericho benchmark games, including Zork1, XTX significantly outperforms strong baselines in both deterministic and stochastic settings, with ablations confirming the value of explicit stage separation and policy mixing. This approach provides a scalable framework for strategic exploration in language-driven environments and informs future designs for directed exploration in large, evolving action spaces.

Abstract

Text adventure games present unique challenges to reinforcement learning methods due to their combinatorially large action spaces and sparse rewards. The interplay of these two factors is particularly demanding because large action spaces require extensive exploration, while sparse rewards provide limited feedback. This work proposes to tackle the explore-vs-exploit dilemma using a multi-stage approach that explicitly disentangles these two strategies within each episode. Our algorithm, called eXploit-Then-eXplore (XTX), begins each episode using an exploitation policy that imitates a set of promising trajectories from the past, and then switches over to an exploration policy aimed at discovering novel actions that lead to unseen state spaces. This policy decomposition allows us to combine global decisions about which parts of the game space to return to with curiosity-based local exploration in that space, motivated by how a human may approach these games. Our method significantly outperforms prior approaches by 27% and 11% average normalized score over 12 games from the Jericho benchmark (Hausknecht et al., 2020) in both deterministic and stochastic settings, respectively. On the game of Zork1, in particular, XTX obtains a score of 103, more than a 2x improvement over prior methods, and pushes past several known bottlenecks in the game that have plagued previous state-of-the-art methods.

Figures (19)

• Figure 1: Sample game paths and state observations from Zork1. Starting from the leftmost state ('West of House'), the agent encounters several novel and unique valid actions (e.g Odysseus, Echo) (in brown) across different states in the game. In order to make progress, our algorithm (XTX) strategically re-visits different frontiers in the state space (red and blue circles) and performs strategic local exploration to overcome bottleneck states (e.g. 'Troll Room') and dead-ends (e.g. 'Cellar'). Solid borders indicate visited states, dotted ones indicate potential future states.
• Figure 2: Average episode scores for 4 ablation models across 6 games. Overall, we find both the strategic inverse dynamics policy and the explicit exploitation policy to be key for our algorithm.
• Figure 3: Average episode scores for 4 ablation models across 12 games. Overall, we find both the strategic inverse dynamics policy and the explicit exploitation policy to be key for our algorithm. Scores for dragon were clipped to be between 0 and 1.
• Figure 4: Average episode score throughout training for all ablations on Zork1. Shaded areas indicate one standard deviation.
• Figure 5: Average episode score throughout training for all ablations on Inhumane. Shaded areas indicate one standard deviation.