Large Language Models Think Too Fast To Explore Effectively

TL;DR

The paper investigates how large language models explore in open-ended tasks using Little Alchemy 2 as a testbed. It contrasts uncertainty-driven exploration with empowerment-based exploration, employing regression analyses, thought tracing, and sparse autoencoder mappings to understand internal representations. The findings show most LLMs underperform humans, except for o1, while DeepSeek-R1 achieves human-like exploration through deeper, iterative reasoning. The work highlights a temporal mismatch in how LLMs represent empowerment and uncertainty, and argues for architecture- and training-level changes to enable more effective exploration with practical implications for adaptive AI systems.

Abstract

Large Language Models (LLMs) have emerged with many intellectual capacities. While numerous benchmarks assess their intelligence, limited attention has been given to their ability to explore--an essential capacity for discovering new information and adapting to novel environments in both natural and artificial systems. The extent to which LLMs can effectively explore, particularly in open-ended tasks, remains unclear. This study investigates whether LLMs can surpass humans in exploration during an open-ended task, using Little Alchemy 2 as a paradigm, where agents combine elements to discover new ones. Results show most LLMs underperform compared to humans, except for the o1 model, with traditional LLMs relying primarily on uncertainty-driven strategies, unlike humans who balance uncertainty and empowerment. Results indicate that traditional reasoning-focused LLMs, such as GPT-4o, exhibit a significantly faster and less detailed reasoning process, limiting their exploratory performance. In contrast, the DeepSeek reasoning model demonstrates prolonged, iterative thought processes marked by repetitive analysis of combinations and past trials, reflecting a more thorough and human-like exploration strategy. Representational analysis of the models with Sparse Autoencoders (SAE) revealed that uncertainty and choices are represented at earlier transformer blocks, while empowerment values are processed later, causing LLMs to think too fast and make premature decisions, hindering effective exploration. These findings shed light on the limitations of LLM exploration and suggest directions for improving their adaptability.

Figures (16)

• Figure 1: A: LLMs Game Process. LLMs select two elements per trial based on the inventory and trial history. B: Human Game Interface. Players select two elements to discover new elements, added to the inventory. C: LLMs and Human Performance.
• Figure 2: Human and LLMs different Temperatures' Performance. LLM and Human Performance Across Temperatures. For LLMs, we set four temperatures (0, 0.3, 0.7, 1). LLMs (GPT-4o, LLaMA3.1-8B, LLaMA3.1-70B) achieve their best performance at $\text{temperature} = 1$.
• Figure 3: Regression Estimates by Temperature and Model. All models show lower empowerment weights than humans, except o1. As temperature increases, uncertainty weights rise, with o1 showing the highest weights across all models and humans.
• Figure 4: Comparison of reasoning depth and token usage between DeepSeek-R1 and GPT-4o.A: Per-Trial Reasoning Depth. DeepSeek-R1 shows substantially longer reasoning sequences and consistently employs all reasoning labels, while GPT-4o exhibits significantly shorter sequences and fewer reasoning types. B: Per-trial Token Usage by Reasoning Labels. DeepSeek-R1 allocates a significantly higher number of tokens across all reasoning labels, especially emphasizing outcome_prediction and combination_analysis. GPT-4o uses substantially fewer tokens, limiting the depth and breadth of analysis.
• Figure 5: SAE Correlation Analysis. Maximum correlation of uncertainty values across layers, peaking at layer 2. Maximum correlation of empowerment values across layers, peaking at layer 72. Maximum beta weight of choices across layers, peaking at layer 1.