Development of Cognitive Intelligence in Pre-trained Language Models

TL;DR

This work investigates whether ten pre-trained language models exhibit cognitive and developmental alignment with human psychometric intelligence by evaluating intermediate training checkpoints across four domains: numeric abilities, linguistic abilities, conceptual understanding, and fluid reasoning. Using a psychometric-guided task suite and multiple model families, it reveals a consistent window of maximal developmental alignment during training, with structure arising early and subsequent training optimizing loss rather than cognition. Scaling generally enhances linguistic and fluid reasoning, while numeric alignment shows a robust distance effect but weaker ratio sensitivity. The findings support using PLMs as cognitive science models under carefully chosen developmental tasks, while emphasizing the importance of developmental trajectories and the limits of final-model comparisons for cognitive modeling.

Abstract

Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). The increasing cognitive alignment of these models has made them candidates for cognitive science theories. Prior research into the emergent cognitive abilities of PLMs has largely been path-independent to model training, i.e., has focused on the final model weights and not the intermediate steps. However, building plausible models of human cognition using PLMs would benefit from considering the developmental alignment of their performance during training to the trajectories of children's thinking. Guided by psychometric tests of human intelligence, we choose four sets of tasks to investigate the alignment of ten popular families of PLMs and evaluate their available intermediate and final training steps. These tasks are Numerical ability, Linguistic abilities, Conceptual understanding, and Fluid reasoning. We find a striking regularity: regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development. Before that window, training appears to endow "blank slate" models with the requisite structure to be poised to rapidly learn from experience. After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.

Figures (7)

• Figure 1: A list of cognitive intelligence tasks under consideration.
• Figure 2: Mental Number Line: Organization of magnitude representations in a logarithmically scaled manner.
• Figure 3: Example adaptation of visual RPM problems to the textual format. Each image is decomposed into tuples of (type, size, color). Type indicates the shape of the image.
• Figure 4: Developmental trajectory of the Pythia suite of models on the psychometric intelligence tasks as a function number of tokens seen. We display the x-axis in a log-scaled manner as maximal development occurs in the range of 100 Million to 20 Billion tokens seen for all tasks. The windows of maximal development are illustrated by the blue shading.
• Figure 5: Development of the idea of "numbers" in Pythia. The y-axis indicates the maximum cosine similarity between the latent representations of any two number words/ digits.