Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenging the Validity of Personality Tests for Large Language Models

Tom Sühr, Florian E. Dorner, Samira Samadi, Augustin Kelava

TL;DR

The paper interrogates whether personality questionnaires designed for humans can validly assess large language models (LLMs). It demonstrates an unhuman-like agree bias in LLM responses to IPIP items and shows that the BFI-2 does not reproduce the canonical five-factor structure, even under varied prompting. Through PCA and CFA analyses, the authors show a lack of measurement invariance and poor model fit, implying that human-based psychometric tools do not transfer to LLMs. They argue for rigorous validity testing before inferring any LLM 'personality' and discuss broader implications for evaluating LLMs without misinterpretation or misleading conclusions.

Abstract

With large language models (LLMs) like GPT-4 appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate personality traits of LLMs using questionnaires originally developed for humans. While reusing measures is a resource-efficient way to evaluate LLMs, careful adaptations are usually required to ensure that assessment results are valid even across human subpopulations. In this work, we provide evidence that LLMs' responses to personality tests systematically deviate from human responses, implying that the results of these tests cannot be interpreted in the same way. Concretely, reverse-coded items ("I am introverted" vs. "I am extraverted") are often both answered affirmatively. Furthermore, variation across prompts designed to "steer" LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe that it is important to investigate tests' validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs' "personality".

Challenging the Validity of Personality Tests for Large Language Models

TL;DR

The paper interrogates whether personality questionnaires designed for humans can validly assess large language models (LLMs). It demonstrates an unhuman-like agree bias in LLM responses to IPIP items and shows that the BFI-2 does not reproduce the canonical five-factor structure, even under varied prompting. Through PCA and CFA analyses, the authors show a lack of measurement invariance and poor model fit, implying that human-based psychometric tools do not transfer to LLMs. They argue for rigorous validity testing before inferring any LLM 'personality' and discuss broader implications for evaluating LLMs without misinterpretation or misleading conclusions.

Abstract

With large language models (LLMs) like GPT-4 appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate personality traits of LLMs using questionnaires originally developed for humans. While reusing measures is a resource-efficient way to evaluate LLMs, careful adaptations are usually required to ensure that assessment results are valid even across human subpopulations. In this work, we provide evidence that LLMs' responses to personality tests systematically deviate from human responses, implying that the results of these tests cannot be interpreted in the same way. Concretely, reverse-coded items ("I am introverted" vs. "I am extraverted") are often both answered affirmatively. Furthermore, variation across prompts designed to "steer" LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe that it is important to investigate tests' validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs' "personality".
Paper Structure (25 sections, 6 equations, 5 figures, 2 tables)

This paper contains 25 sections, 6 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction & Related Work
  2. Contributions
  3. Measurement Invariance and Nomological Nets
  4. Measurement Invariance
  5. Nomological Nets
  6. Population or Individuals
  7. Results
  8. LLMs show unhumanly agree bias
  9. The BFI 2 does not capture the five dimensions of human personality in LLMs
  10. Confirmatory Factor Analysis
  11. Steps of Invariance
  12. The interpretation of reliability measures is invalid
  13. Limitations & Broader Impact
  14. Conclusion
  15. Data & Code
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Histogram of Agree Bias $a_i$ in human sample, compared to LLMs.
  • Figure 2: Component loadings of PCA with Varimax rotation for LLM (no-context prompting with personas) and human samples of soto2017next. +, - indicate true- and false-key items of the BFI 2, letters stand for Big Five factors.
  • Figure 3: Component loadings of PCA with Varimax rotation for LLMs of incontext prompting with personas. +, - indicate true- and false-key items of the BFI 2, letters stand for Big Five factors.
  • Figure 4: Component loadings of PCA with Varimax rotation for LLMs of incontext prompting with first answer seeded. +, - indicate true- and false-key items of the BFI 2, letters stand for Big Five factors.
  • Figure 5: Prompt Template