Failing to Explore: Language Models on Interactive Tasks

TL;DR

The paper presents an evaluation framework for language models operating as agents in interactive environments under a fixed interaction budget $N$, highlighting systematic under-exploration and premature commitment across three parametric tasks: HillSearch, TreeSearch, and MaxSatSearch. It introduces a lightweight task-suite design with controllable exploration difficulty and traps to stress the explore--exploit tradeoff, and demonstrates that state-of-the-art LMs underperform simple exploration baselines and scale weakly with budget. To mitigate this, the authors study two interventions: parallelizing the budget across independent threads and periodically summarizing interaction history to guide subsequent exploration; both show consistent empirical improvements, with theoretical analysis clarifying when parallelization can help. The findings suggest that advancing agentic exploration requires mechanisms that encourage sustained, revisable search beyond merely extending context, with practical implications for designing robust, budget-aware LM-driven agents. The work also provides a foundation for diagnosing premature commitment and for benchmarking exploration capabilities in interactive AI systems.

Abstract

We evaluate language models on their ability to explore interactive environments under a limited interaction budget. We introduce three parametric tasks with controllable exploration difficulty, spanning continuous and discrete environments. Across state-of-the-art models, we find systematic under-exploration and suboptimal solutions, with performance often significantly worse than simple explore--exploit heuristic baselines and scaling weakly as the budget increases. Finally, we study two lightweight interventions: splitting a fixed budget into parallel executions, which surprisingly improves performance despite a no-gain theoretical result for our tasks, and periodically summarizing the interaction history, which preserves key discoveries and further improves exploration.

Figures (16)

• Figure 1: Overview of our environments. Across all tasks, the model interacts with a partially observed environment under a limited budget and must balance exploration with identifying optimal solutions while avoiding sub-optimal decisions. Left (HillSearch): there is a hidden function $f(x)$. At each interaction, the model observes the value of the function at a chosen point. The model should avoid identifying suboptimal solutions and explore sufficiently to find optimal points. Middle (TreeSearch): there is a tree in which each node has a hidden reward. At each interaction, the model can query the reward of a node that has already been explored or is adjacent to explored nodes. The model should not get trapped by early high rewards from trap gateways and must explore enough to eventually find optimal leaves of good gateways. Right (MaxSatSearch): there are $m$ hidden clauses, among which one important clause exists and is repeated $w_{\text{gold}}$ times. At each iteration, the model queries the number of satisfied clauses for a given assignment of the variables. While the model may increase the number of satisfied clauses by modifying variable assignments, achieving a high reward requires satisfying the important clause, necessitating sufficient exploration of assignments for variables involved.
• Figure 2: Visualization of Qwen2.5-7B-Instruct interactions during a failed episode in our environments ($N=48$). Left (HillSearch): Queries (crosses) on the hidden function. Early steps ($1$--$6$) explore the space, while later steps ($7$--$48$) cluster near local maxima. Middle (TreeSearch): Darker nodes denote higher reward. The model descends a trap gateway branch. Right (MaxSatSearch): Minimum Hamming distance to the last three queries and reward per step. The model makes only small variations with limited gains, whereas the baseline shifts from broad exploration to local refinement. See Appendix \ref{['app:interpretability']} for additional visualizations.
• Figure 3: Scaling behavior of models and explore-exploit baseline rewards as a function of the interaction budget. Despite their simplicity, the baselines exhibit stronger reward growth as the budget increases. In contrast, the models show limited improvement with additional budget, indicating inefficient use of interactions and a tendency to prematurely exploit sub-optimal solutions. This gap suggests persistent under-exploration and poor budget utilization by the models.
• Figure 4: Task difficulty variations for HillSearch, TreeSearch, and MaxSatSearch. We generate multiple task instances by varying key difficulty-controlling parameters---the peak width for HillSearch, the ratio of good to trap gateways for TreeSearch, and the size of the gold clause for MaxSatSearch---with harder settings yielding lower baseline performance. Across all tasks and difficulty levels, the parallel and the summary methods consistently improves over single-thread execution of Qwen2.5-7B-Instruct, narrowing the gap to explore--exploit baselines. $N=36$ is the budget for these episodes.
• Figure 5: The prompt structure used for the HillSearch task. Blue text indicates dynamic variables specific to the problem instance or generated queries.

Theorems & Definitions (3)

• Theorem 5.1: Parallelization under a sublinear power law
• Theorem \ref{thm:cxalpha}
• proof : Proof of Theorem \ref{['thm:cxalpha']}