Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Influence of Human-inspired Agentic Sophistication in LLM-driven Strategic Reasoners

Vince Trencsenyi, Agnieszka Mensfelt, Kostas Stathis

TL;DR

This work investigates how human-inspired agentic sophistication influences LLM-driven strategic reasoning in two-player guessing games. It deploys a formal multi-agent simulation (OODA-inspired) with an umpire, contexts, and instruction templates to compare simple, reasoning-based, and LLM-based agents against human data, including out-of-distribution scenarios. The study combines an EWA baseline with two Claude LLMs (Haiku and Sonnet) across varied prompting configurations and a MoA guidance mechanism, evaluating performance at population and subpopulation levels using k-level reasoning, distributional similarity, and equilibrium play metrics. Findings show that human-like cognitive structures can improve alignment with human strategizing, but the relationship between architectural complexity and human-likeness is non-linear and model-dependent; simpler configurations can generalize better, while larger LLMs may overfit training priors. The results inform design choices for agentic AI in game-theoretic and human-centric contexts, emphasizing model- and task-specific matching over a default push toward maximal sophistication.

Abstract

The rapid rise of large language models (LLMs) has shifted artificial intelligence (AI) research toward agentic systems, motivating the use of weaker and more flexible notions of agency. However, this shift raises key questions about the extent to which LLM-based agents replicate human strategic reasoning, particularly in game-theoretic settings. In this context, we examine the role of agentic sophistication in shaping artificial reasoners' performance by evaluating three agent designs: a simple game-theoretic model, an unstructured LLM-as-agent model, and an LLM integrated into a traditional agentic framework. Using guessing games as a testbed, we benchmarked these agents against human participants across general reasoning patterns and individual role-based objectives. Furthermore, we introduced obfuscated game scenarios to assess agents' ability to generalise beyond training distributions. Our analysis, covering over 2000 reasoning samples across 25 agent configurations, shows that human-inspired cognitive structures can enhance LLM agents' alignment with human strategic behaviour. Still, the relationship between agentic design complexity and human-likeness is non-linear, highlighting a critical dependence on underlying LLM capabilities and suggesting limits to simple architectural augmentation.

The Influence of Human-inspired Agentic Sophistication in LLM-driven Strategic Reasoners

TL;DR

This work investigates how human-inspired agentic sophistication influences LLM-driven strategic reasoning in two-player guessing games. It deploys a formal multi-agent simulation (OODA-inspired) with an umpire, contexts, and instruction templates to compare simple, reasoning-based, and LLM-based agents against human data, including out-of-distribution scenarios. The study combines an EWA baseline with two Claude LLMs (Haiku and Sonnet) across varied prompting configurations and a MoA guidance mechanism, evaluating performance at population and subpopulation levels using k-level reasoning, distributional similarity, and equilibrium play metrics. Findings show that human-like cognitive structures can improve alignment with human strategizing, but the relationship between architectural complexity and human-likeness is non-linear and model-dependent; simpler configurations can generalize better, while larger LLMs may overfit training priors. The results inform design choices for agentic AI in game-theoretic and human-centric contexts, emphasizing model- and task-specific matching over a default push toward maximal sophistication.

Abstract

The rapid rise of large language models (LLMs) has shifted artificial intelligence (AI) research toward agentic systems, motivating the use of weaker and more flexible notions of agency. However, this shift raises key questions about the extent to which LLM-based agents replicate human strategic reasoning, particularly in game-theoretic settings. In this context, we examine the role of agentic sophistication in shaping artificial reasoners' performance by evaluating three agent designs: a simple game-theoretic model, an unstructured LLM-as-agent model, and an LLM integrated into a traditional agentic framework. Using guessing games as a testbed, we benchmarked these agents against human participants across general reasoning patterns and individual role-based objectives. Furthermore, we introduced obfuscated game scenarios to assess agents' ability to generalise beyond training distributions. Our analysis, covering over 2000 reasoning samples across 25 agent configurations, shows that human-inspired cognitive structures can enhance LLM agents' alignment with human strategic behaviour. Still, the relationship between agentic design complexity and human-likeness is non-linear, highlighting a critical dependence on underlying LLM capabilities and suggesting limits to simple architectural augmentation.
Paper Structure (29 sections, 13 equations, 6 figures, 8 tables)

This paper contains 29 sections, 13 equations, 6 figures, 8 tables.

Table of Contents

  1. Human Reasoning in 2-Player Guessing Games
  2. Methods
  3. Multi-agent Simulation Framework
  4. Framework Overview
  5. Agent Processes
  6. Game Formalisation
  7. Roadmap
  8. Experience Weighted Attraction
  9. Artificial Reasoners
  10. LLM
  11. Prompting Model
  12. Agentic Concept
  13. Experiments and Discussion
  14. Benchmark Model Performance
  15. Artifical Reasoners
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Distribution of human guesses per student and expert cohort. Vertical grey lines mark the standard k-levels for guessing games.
  • Figure 2: Distribution of EWA over joint human results and k-levels.
  • Figure 3: Heatmap showing the absolute distance between agents' and humans' mean reasoning sophistication according to k-levels.
  • Figure 4: KDEs of $S$ (top) and $R$ (bottom) agents with $C$,$M$ variations. Red and blue correspond to overall results, and lighter/darker shades correspond to specific cohorts.
  • Figure 5: Delta error between agents' and humans' mean k-level difference for students and experts with significance levels: * p < 0.05, ** p < 0.01, *** p < 0.001.
  • ...and 1 more figures