Evidence for Limited Metacognition in LLMs
Christopher Ackerman
TL;DR
This paper presents two game-based paradigms, the Delegate Game and the Second Chance Game, to quantify metacognition in large language models without relying on self-reports. Using a range of frontier and near-frontier models across two datasets, the authors show that recent models exhibit only limited, context-dependent metacognitive abilities, evidenced by weak but significant links between internal signals (as proxied by token probabilities and baseline correctness) and delegation or self-modeling decisions. The findings suggest that internal confidence signals exist but are not robust enough to consistently outperform non-introspective cues, and that post-training regimens may influence metacognitive abilities. Overall, the work provides a quantitative framework for assessing metacognition in LLMs, highlights the nuanced differences across models, and points to future directions for interpretability, human benchmarking, and comprehensive metacognition components.
Abstract
The possibility of LLM self-awareness and even sentience is gaining increasing public attention and has major safety and policy implications, but the science of measuring them is still in a nascent state. Here we introduce a novel methodology for quantitatively evaluating metacognitive abilities in LLMs. Taking inspiration from research on metacognition in nonhuman animals, our approach eschews model self-reports and instead tests to what degree models can strategically deploy knowledge of internal states. Using two experimental paradigms, we demonstrate that frontier LLMs introduced since early 2024 show increasingly strong evidence of certain metacognitive abilities, specifically the ability to assess and utilize their own confidence in their ability to answer factual and reasoning questions correctly and the ability to anticipate what answers they would give and utilize that information appropriately. We buttress these behavioral findings with an analysis of the token probabilities returned by the models, which suggests the presence of an upstream internal signal that could provide the basis for metacognition. We further find that these abilities 1) are limited in resolution, 2) emerge in context-dependent manners, and 3) seem to be qualitatively different from those of humans. We also report intriguing differences across models of similar capabilities, suggesting that LLM post-training may have a role in developing metacognitive abilities.