Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experiments

TL;DR

The paper probes whether LLMs exhibit human-like exploration–exploitation (E&E) strategies in sequential decision tasks by contrasting them with humans and traditional MAB algorithms in stationary and non-stationary environments. Using two canonical MAB setups and interpretable choice models grounded in Bayesian learning, it quantifies directed versus random exploration and how thinking traces influence LLM decision-making. The study finds that enabling thinking shifts LLM behavior toward a mix of exploration types more reminiscent of humans, improving performance in simple settings but leaving gaps in adaptive directed exploration under non-stationary, higher-armed conditions. The results underscore both the potential and the limits of LLMs as human simulators for dynamic decision-making and point to design directions (prompt design, internal thinking control) to enhance their adaptive E&E capabilities in complex tasks.

Abstract

Large language models (LLMs) are increasingly used to simulate or automate human behavior in complex sequential decision-making settings. A natural question is then whether LLMs exhibit similar decision-making behavior to humans, and can achieve comparable (or superior) performance. In this work, we focus on the exploration-exploitation (E&E) tradeoff, a fundamental aspect of dynamic decision-making under uncertainty. We employ canonical multi-armed bandit (MAB) experiments introduced in the cognitive science and psychiatry literature to conduct a comparative study of the E&E strategies of LLMs, humans, and MAB algorithms. We use interpretable choice models to capture the E&E strategies of the agents and investigate how enabling thinking traces, through both prompting strategies and thinking models, shapes LLM decision-making. We find that enabling thinking in LLMs shifts their behavior toward more human-like behavior, characterized by a mix of random and directed exploration. In a simple stationary setting, thinking-enabled LLMs exhibit similar levels of random and directed exploration compared to humans. However, in more complex, non-stationary environments, LLMs struggle to match human adaptability, particularly in effective directed exploration, despite achieving similar regret in certain scenarios. Our findings highlight both the promise and limits of LLMs as simulators of human behavior and tools for automated decision-making and point to potential areas for improvement.

Figures (12)

• Figure 1: Illustration of the prompt protocol and interaction flow in the 2-armed bandit experiment.
• Figure 2: Visualizing the SM-3 model parameters across LLMs in the 2-armed setting
• Figure 3: Visualizing the SM-3 model parameters across thinking models in the 2-armed setting
• Figure 4: Visualizing the SM-3 model parameters across basic models in the 4-armed setting
• Figure 5: Visualizing the SM-3 model parameters across thinking models in the 4-armed setting