The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories

TL;DR

The paper assesses the viability of using pre-trained language models (PLMs) as cognitive and developmental theories, arguing for a three-stage mapping from human data to model outputs: adapt stimuli, apply a linking hypothesis, and compare model and human performance. It catalogs pitfalls as commission and omission, highlighting distal linking hypotheses, interpretability gaps, and developmental-context neglect, and proposes explicit criteria for credible use, including empirical triangulation, path-dependency assessment, and cross-task correlations. It surveys linking-hypothesis approaches—similarity computations, surprisal, and prompting—and discusses their advantages and limitations for cognitive modeling. The authors advocate development-aware practices, such as developmentally plausible pre-training data and core-task tuning prior to broader evaluation, to improve developmental alignment and predictive utility across tasks. Collectively, the work provides a structured framework to harness PLMs for cognitive science while emphasizing caution, transparency, and ongoing methodological refinement.

Abstract

Many studies have evaluated the cognitive alignment of Pre-trained Language Models (PLMs), i.e., their correspondence to adult performance across a range of cognitive domains. Recently, the focus has expanded to the developmental alignment of these models: identifying phases during training where improvements in model performance track improvements in children's thinking over development. However, there are many challenges to the use of PLMs as cognitive science theories, including different architectures, different training data modalities and scales, and limited model interpretability. In this paper, we distill lessons learned from treating PLMs, not as engineering artifacts but as cognitive science and developmental science models. We review assumptions used by researchers to map measures of PLM performance to measures of human performance. We identify potential pitfalls of this approach to understanding human thinking, and we end by enumerating criteria for using PLMs as credible accounts of cognition and cognitive development.

Figures (1)

• Figure 1: The three-stage mapping between human data and model performance when establishing the sufficiency and utility of PLMs as cognitive and developmental science models.