Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires

Pranav Bhandari, Usman Naseem, Amitava Datta, Nicolas Fay, Mehwish Nasim

TL;DR

Problem: Do LLMs exhibit human-like personality traits, and can psychometric tools meaningfully quantify them without data leakage? Approach: The authors apply restructured Big Five questionnaires to multiple LLMs (e.g., GPT-4, GPT-4o-mini, Llama variants), use randomized prompt sequences, and compute consistency via the coefficient of variation across 100 runs. Contributions: They report distinct personality profiles across models, with higher Openness, Conscientiousness, and Agreeableness, and model-type-specific dimensional dominance (e.g., Agreeableness in OpenAI models; Openness or Conscientiousness in Llama variants). Significance: The study provides a rigorous framework for interpreting LLM behavior through psychometrics and highlights the need for careful questionnaire design to avoid contamination and misinterpretation.

Abstract

Psychological assessment tools have long helped humans understand behavioural patterns. While Large Language Models (LLMs) can generate content comparable to that of humans, we explore whether they exhibit personality traits. To this end, this work applies psychological tools to LLMs in diverse scenarios to generate personality profiles. Using established trait-based questionnaires such as the Big Five Inventory and by addressing the possibility of training data contamination, we examine the dimensional variability and dominance of LLMs across five core personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Our findings reveal that LLMs exhibit unique dominant traits, varying characteristics, and distinct personality profiles even within the same family of models.

Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires

TL;DR

Problem: Do LLMs exhibit human-like personality traits, and can psychometric tools meaningfully quantify them without data leakage? Approach: The authors apply restructured Big Five questionnaires to multiple LLMs (e.g., GPT-4, GPT-4o-mini, Llama variants), use randomized prompt sequences, and compute consistency via the coefficient of variation across 100 runs. Contributions: They report distinct personality profiles across models, with higher Openness, Conscientiousness, and Agreeableness, and model-type-specific dimensional dominance (e.g., Agreeableness in OpenAI models; Openness or Conscientiousness in Llama variants). Significance: The study provides a rigorous framework for interpreting LLM behavior through psychometrics and highlights the need for careful questionnaire design to avoid contamination and misinterpretation.

Abstract

Psychological assessment tools have long helped humans understand behavioural patterns. While Large Language Models (LLMs) can generate content comparable to that of humans, we explore whether they exhibit personality traits. To this end, this work applies psychological tools to LLMs in diverse scenarios to generate personality profiles. Using established trait-based questionnaires such as the Big Five Inventory and by addressing the possibility of training data contamination, we examine the dimensional variability and dominance of LLMs across five core personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Our findings reveal that LLMs exhibit unique dominant traits, varying characteristics, and distinct personality profiles even within the same family of models.

Paper Structure

This paper contains 12 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions:
  3. Literature Review:
  4. Methodology
  5. Removing Training Data Contamination
  6. Systematic Sequence of Prompts
  7. Calculate Coefficient of Variation
  8. Results
  9. Personality Profiles of LLMs
  10. Dimensional Variability Across Models
  11. Dimensional dominance by Model Type
  12. Conclusion

Figures (2)

  • Figure 1: Methodology involves combining system and user prompts with questionnaires to administer to LLMs, extracting, processing and plotting the responses respectively.
  • Figure 2: Radar plots to visualise all five personality questionnaires across their dimensions, comparing results from all LLMs. While scores are generally similar, notable gaps in specific personality traits are evident.