Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Limits of Emergent Reasoning of Large Language Models in Agentic Frameworks for Deterministic Games

Chris Su, Harrison Li, Matheus Marques, George Flint, Kevin Zhu, Sunishchal Dev

TL;DR

This work probes whether emergent reasoning in large language models truly transcends training data and architectural limitations by externalizing state tracking in an agentic Tower of Hanoi setting. It compares a traditional one-shot baseline to an interactive environment where the model acts stepwise via tool calls, and it analyzes the model's decision policies using Q-values and Jensen-Shannon divergence. Across both setups, the authors observe persistent performance collapse, with the agentic framework not preventing, and often accelerating, failure, plus evidence of deterministic looping and divergence from both optimal and random policies. These findings challenge the view that scaling alone yields robust emergent reasoning, highlighting the need for evaluation frameworks and architectures that support dynamic state management and planning.

Abstract

Recent work reports that Large Reasoning Models (LRMs) undergo a collapse in performance on solving puzzles beyond certain perplexity thresholds. In subsequent discourse, questions have arisen as to whether the nature of the task muddles an evaluation of true reasoning. One potential confound is the requirement that the model keep track of the state space on its own. We provide a large language model (LLM) with an environment interface for Tower of Hanoi problems, allowing it to make a move with a tool call, provide written justification, observe the resulting state space, and reprompt itself for the next move. We observe that access to an environment interface does not delay or eradicate performance collapse. Furthermore, LLM-parameterized policy analysis reveals increasing divergence from both optimal policies and uniformly random policies, suggesting that the model exhibits mode-like collapse at each level of complexity, and that performance is dependent upon whether the mode reflects the correct solution for the problem. We suggest that a similar phenomena might take place in LRMs.

Limits of Emergent Reasoning of Large Language Models in Agentic Frameworks for Deterministic Games

TL;DR

This work probes whether emergent reasoning in large language models truly transcends training data and architectural limitations by externalizing state tracking in an agentic Tower of Hanoi setting. It compares a traditional one-shot baseline to an interactive environment where the model acts stepwise via tool calls, and it analyzes the model's decision policies using Q-values and Jensen-Shannon divergence. Across both setups, the authors observe persistent performance collapse, with the agentic framework not preventing, and often accelerating, failure, plus evidence of deterministic looping and divergence from both optimal and random policies. These findings challenge the view that scaling alone yields robust emergent reasoning, highlighting the need for evaluation frameworks and architectures that support dynamic state management and planning.

Abstract

Recent work reports that Large Reasoning Models (LRMs) undergo a collapse in performance on solving puzzles beyond certain perplexity thresholds. In subsequent discourse, questions have arisen as to whether the nature of the task muddles an evaluation of true reasoning. One potential confound is the requirement that the model keep track of the state space on its own. We provide a large language model (LLM) with an environment interface for Tower of Hanoi problems, allowing it to make a move with a tool call, provide written justification, observe the resulting state space, and reprompt itself for the next move. We observe that access to an environment interface does not delay or eradicate performance collapse. Furthermore, LLM-parameterized policy analysis reveals increasing divergence from both optimal policies and uniformly random policies, suggesting that the model exhibits mode-like collapse at each level of complexity, and that performance is dependent upon whether the mode reflects the correct solution for the problem. We suggest that a similar phenomena might take place in LRMs.
Paper Structure (19 sections, 30 equations, 6 figures)

This paper contains 19 sections, 30 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Tower of Hanoi
  5. Experimental Setup
  6. Baseline
  7. Environment-Based Agentic Framework
  8. LLM-Parameterized Policy Analysis
  9. Results
  10. Baseline
  11. Agentic Framework
  12. LLM-Parameterized Policy Analysis
  13. Discussion
  14. Limitations.
  15. Conclusion
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Agentic framework is a closed loop interaction between the agent, environment, and game validator.
  • Figure 2: Comparison of success rates of LLM and LRM one-shot generation: (Left) Claude 3.7 Sonnet with and without "thinking" mechanism, (Right) DeepSeek V3.1 vs R1. Line charts display success rate as a function of puzzle complexity.
  • Figure 3: Success rate of models in an agentic framework (Claude 3.7 Sonnet + environment, DeepSeek V3.1 + environment) in comparison to the baseline (Claude 3.7 Sonnet, DeepSeek V3.1) at increasing complexity levels.
  • Figure 4: Loop rate of the models in an agentic framework (Claude 3.7 Sonnet + environment, DeepSeek V3.1 + environment) at increasing complexity levels.
  • Figure 5: Proportion of unique length $k$ transitions taken from state $s$, given that $s$ was visited by the model at least twice. Lower values mean that the model takes less unique length $k$ trajectories. These graphs are similar since every $k=3$ subsequence from $s$ also contains the $k=2$ subsequence from $s$.
  • ...and 1 more figures