Is Self-knowledge and Action Consistent or Not: Investigating Large Language Model's Personality

TL;DR

This work investigates whether LLMs' self-reported personality meaningfully corresponds to their actual behavior by constructing a bilingual, 180-item personality-knowledge and 180-item behavior-tendency corpus grounded in the Big Five and MBTI frameworks. Using rigorous reliability and consented human validation, the authors compare LLM responses to human response patterns across self-knowledge and action, employing multiple similarity and consistency metrics. Humans exhibit strong self-knowledge-action congruence, while several LLMs show notable gaps, suggesting that current models only partially mimic human personality dynamics. The study advances psychometric evaluation for LLMs and highlights implications for safer, more reliable, and psychologically realistic AI-human interactions.

Abstract

In this study, we delve into the validity of conventional personality questionnaires in capturing the human-like personality traits of Large Language Models (LLMs). Our objective is to assess the congruence between the personality traits LLMs claim to possess and their demonstrated tendencies in real-world scenarios. By conducting an extensive examination of LLM outputs against observed human response patterns, we aim to understand the disjunction between self-knowledge and action in LLMs.