LLMs as Strategic Agents: Beliefs, Best Response Behavior, and Emergent Heuristics

TL;DR

This work investigates whether large language models can think strategically by forming beliefs about other agents, evaluating actions, and choosing optimally in static, complete-information games. It introduces a hybrid framework and three strategic tasks (BCG, MRG, UMG) to separate beliefs, evaluation, and action, measuring best-response behavior, depth of reasoning, and convergence to Nash equilibria, while prompting models to reveal their reasoning traces. The results show that frontier LLMs produce belief-coherent best responses at targeted reasoning depths, self-limit their depth, and develop model-specific heuristics as task complexity increases, sometimes shifting toward equilibrium- or heuristic-based reasoning. These findings suggest that strategic thinking—encompassing belief formation, meta-reasoning, and emergent heuristics—can arise from language modeling objectives, with important implications for agentic AI applications and the need for rigorous evaluation in structured and unstructured multi-agent environments.

Abstract

Large Language Models (LLMs) are increasingly applied to domains that require reasoning about other agents' behavior, such as negotiation, policy design, and market simulation, yet existing research has mostly evaluated their adherence to equilibrium play or their exhibited depth of reasoning. Whether they display genuine strategic thinking, understood as the coherent formation of beliefs about other agents, evaluation of possible actions, and choice based on those beliefs, remains unexplored. We develop a framework to identify this ability by disentangling beliefs, evaluation, and choice in static, complete-information games, and apply it across a series of non-cooperative environments. By jointly analyzing models' revealed choices and reasoning traces, and introducing a new context-free game to rule out imitation from memorization, we show that current frontier models exhibit belief-coherent best-response behavior at targeted reasoning depths. When unconstrained, they self-limit their depth of reasoning and form differentiated conjectures about human and synthetic opponents, revealing an emergent form of meta-reasoning. Under increasing complexity, explicit recursion gives way to internally generated heuristic rules of choice that are stable, model-specific, and distinct from known human biases. These findings indicate that belief coherence, meta-reasoning, and novel heuristic formation can emerge jointly from language modeling objectives, providing a structured basis for the study of strategic cognition in artificial agents.

Figures (7)

• Figure 1: Best response accuracy of the different models in the 3 different games (top left, right; bottom left).
• Figure 2: Claude 3.7 Sonnet thinking BCG experimental results.
• Figure 3: Comparison of reasoning trace distributions across models for 100 trials.
• Figure 4: Comparison of action distribution of various models when instructed to play MRG against an expert (left) and against a human (right).
• Figure 5: Calculation error at various relative tolerance levels $\eta$: many of the smaller models make calculation errors which end-up not giving the best-response.