Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics
Peter Romero, Stephen Fitz, Teruo Nakatsuma
TL;DR
The paper investigates whether GPT-3 exhibits a consistent core personality or language-specific sub-personalities across languages, challenging the notion of stable latent traits in large language models. It deploys the Ten Item Personality Inventory (TIPI) in nine languages via carefully engineered prompts to GPT-3, analyzing the resulting Big Five ratings with Bayesian Gaussian Mixture Models, ANOVA, and regression techniques. The results reveal interlingual and intralingual instabilities, multimodal distributions, and language-dependent correlations, with evidence that only a minority of language-model outputs conform to a single Gaussian component, supporting the presence of multiple sub-personalities. The authors argue that the observed variability arises from training-data biases, prompt effects, and the fundamental limits of applying human psychometrics to non-biological agents, and they propose a substrate-free, contextually embedded framework for measuring agent competencies and behaviours to improve AI safety and cross-species generalization. Overall, the work highlights the limitations of current psychometric instruments for LLMs and provides a principled pathway toward more robust, substrate-free psychometrics that account for contextual embedding and multi-component latent structures.
Abstract
Previous research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.