Trapped in the past? Disentangling fluid and crystallized intelligence of large language models using chess
Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsäcker
TL;DR
This work investigates whether large language models leverage crystallized (memory-based) or fluid (first-principles) intelligence when solving problems, using chess as a controlled testbed. It introduces a distributional framework that partitions chess positions into within-distribution (WD), near-distribution (ND), and out-of-distribution (OOD) categories based on training data proximity and evaluates multiple GPT generations under varying reasoning regimes, benchmarking moves against Stockfish. The analysis reveals a consistent gradient: model performance declines as fluid-generalization demands rise, with OOD tasks approaching random play, and reasoning augmentation providing benefits that shrink per token as the task becomes less familiar; while newer models improve, gains saturate for tasks outside the training distribution. The results argue that current architectures remain limited in systematic generalization, indicating that scaling and reasoning alone are unlikely to yield robust fluid intelligence in formal domains without new representations or inference mechanisms, with important implications for safety, reliability, and interpretability. $p_{train}(x)$ and $p_{test}(x)$ are central to the framework, and the generalization gap $Delta_gen = E_{x~p_test}[L(f(x))] - E_{x~p_train}[L(f(x))]$ summarizes fluid versus crystallized performance across distributions.
Abstract
Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall (crystallized intelligence) or reasoning ability (fluid intelligence). We introduce chess as a controlled testbed for disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in training corpus proximity--ranging from common states solvable by memorization to novel ones requiring first-principles reasoning. We systematically evaluate multiple GPT generations under varying reasoning intensities. Our analysis reveals a clear gradient: performance consistently degrades as fluid intelligence demands increase. Notably, in out-of-distribution tasks, performance collapses to random levels. While newer models improve, progress slows significantly for tasks outside the training distribution. Furthermore, while reasoning-augmented inference improves performance, its marginal benefit per token decreases with distributional proximity. These results suggest current architectures remain limited in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust fluid intelligence.