Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs

TL;DR

The paper systematically evaluates working memory and executive functions in large language models using a broad task battery, extending beyond traditional n-back to assess simple and complex spans, attentional control, and cognitive flexibility. It finds that LLMs often exceed human WM capacity, with performance strongly tied to model size, but struggle with tasks requiring manipulation, updating, and flexible strategy use. Reasoning-enabled variants show mixed benefits: they can boost attentional control on simple multi-turn tasks but do not reliably improve complex planning or overall WM capacity and can be inefficient. The study argues that WM capacity is not the sole bottleneck for LLM intelligence, highlighting the need to strengthen core executive functions and to develop more efficient reasoning-based strategies rather than indiscriminately scaling models.

Abstract

Working memory, or the ability to hold and manipulate information in the mind, is a critical component of human intelligence and executive functioning. It is correlated with performance on various cognitive tasks, including measures of fluid intelligence, which encompasses reasoning and problem solving. We use a comprehensive set of classic working memory tasks to estimate the working memory capacity of large language models (LLMs). We find that in most cases, LLMs exceed normative human scores. However, we do not find that the increased capacity of working memory is associated with higher performance on other executive functioning tasks or problem solving benchmarks. These results suggest that LLMs may have deficits in attentional control and cognitive flexibility, which result in difficulties with inhibiting automatic responses and adapting to shifting information. Our findings suggest that current reasoning models have mixed results in compensating for these deficits.

Figures (14)

• Figure 1: Working memory is a keystone component of high-level human cognition. We administer a standard set of working memory tasks to LLMs and find that they tend outperform humans, especially in the simplest tasks. However, we find that in tasks of executive function, the LLMs were generally below human baselines.
• Figure 2: Our experimental pipeline.
• Figure 3: Brief overview of the cognitive assessments adapted for LLMs in the present study: (a) working memory tasks, spanning from simple input/output pairs to full dialogues, (b) executive function tasks, ranging relatively simple flanker task to complex Wisconsin Card Sorting.
• Figure 4: We asses both simple span tasks (top) and complex span tasks (bottom). The simple forward digit span task assesses the storage component of short term memory, whereas backward digit span has some overlap with working memory due to the manipulation component. On the other hand, complex span tasks also heavily tax information processing, making them a frequent choice to estimate of working memory spans.
• Figure 5: The Wisconsin Card Sorting Task (WCST). Participants are serially presented with cards to sort, and they must infer an underlying "sorting rule" based on feedback. The rule will periodically change without warning, and people must detect the change and adjust to the new rule. Participants make sorting decisions as quickly as possible. The WCST tests various executive functions, primarily cognitive flexibility as participants adapt from one set of rules to another.