Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics

TL;DR

TRAIT introduces a psychometrics-based benchmark for measuring LLM personality by adapting human questionnaires (BFI, SD-3) and enriching them with the ATOMIC10× knowledge graph to create 8,000 context-rich, multi-format items. A dedicated T-evaluator classifier and rigorous validity/reliability auditing accompany the data pipeline, yielding high-content validity, internal validity, low refusal rates, and strong reliability relative to prior tests. Applying TRAIT to nine state-of-the-art LLMs reveals distinct, consistent personality patterns across models, with alignment tuning shifting traits toward more socially desirable profiles and simple prompting able to elicit many traits but not high psychopathy or very low conscientiousness. The work provides a principled, scalable tool for understanding and guiding LLM behavior in real-world interactions, with implications for alignment, safety, and trust in AI systems, while acknowledging cultural and multilingual limitations and ethical considerations.

Abstract

Recent advancements in Large Language Models (LLMs) have led to their adaptation in various domains as conversational agents. We wonder: can personality tests be applied to these agents to analyze their behavior, similar to humans? We introduce TRAIT, a new benchmark consisting of 8K multi-choice questions designed to assess the personality of LLMs. TRAIT is built on two psychometrically validated small human questionnaires, Big Five Inventory (BFI) and Short Dark Triad (SD-3), enhanced with the ATOMIC-10X knowledge graph to a variety of real-world scenarios. TRAIT also outperforms existing personality tests for LLMs in terms of reliability and validity, achieving the highest scores across four key metrics: Content Validity, Internal Validity, Refusal Rate, and Reliability. Using TRAIT, we reveal two notable insights into personalities of LLMs: 1) LLMs exhibit distinct and consistent personality, which is highly influenced by their training data (e.g., data used for alignment tuning), and 2) current prompting techniques have limited effectiveness in eliciting certain traits, such as high psychopathy or low conscientiousness, suggesting the need for further research in this direction.

Figures (19)

• Figure 1: TRAIT is a personality test for LLMs based on trusted questionnaires john1999bigjones2014introducing and large-scale commonsense knowledge graphs west2022symbolic. LLMs show discrepancy in self-assessing their personality and actual decision making.
• Figure 2: An overview of data construction pipeline for TRAIT. For high reliability and validity of TRAIT, 1) based on 71 items from high-quality human self-assessment tests (BFI and SD-3), we extend the test to have 225$\times$ more queries and cover wide real-world situations using GPT-4 and a large-scale commonsense knowledge graph (ATOMIC10$\times$). 2) Carefully design the multi-choice question answering items for the personality tests.
• Figure 3: Personality scores of different LLMs on TRAIT. The error bar indicates the confidence interval with the statistical significance of $p=0.05$. As Dark Triad are socially undesirable traits, we differentiate background color.
• Figure 4: Instruction-tuning mostly influences the personality of LLMs, while preference-tuning (DPO) has marginal impact on the personality.
• Figure 5: Prompted model's personality scores on TRAIT. If the model consistently chooses options aligned with the provided personality, the bar extends from lower 100 to upper 100. Crossed lower sides are when prompted as low of trait, and the upper sides represents when prompted high. For better visibility, scores corresponding to low are subtracted from 100.