Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifying Multiple Personalities in Large Language Models with External Evaluation

Xiaoyang Song, Yuta Adachi, Jessie Feng, Mouwei Lin, Linhao Yu, Frank Li, Akshat Gupta, Gopala Anumanchipalli, Simerjot Kaur

TL;DR

This paper proposes an external evaluation framework for LLM personality by fine-tuning an MBTI predictor on human MBTI data and applying it to open-ended responses generated by LLMs in two roles (Twitter posts and replies). The authors demonstrate that LLMs exhibit role-dependent personalities, in contrast to humans who show consistent MBTI patterns across roles, highlighting that LLM personality may not be an enduring trait and that human-based personality definitions may not transfer to LLMs. The method combines a three-stage pipeline (model fine-tuning, open-ended response collection, external evaluation with validation on human data) and shows strong per-trait performance (93.3%) with robust overall accuracy (81.0%) for the binary MBTI predictor. The work calls for re-evaluating how personality is defined and measured in LLMs and urges caution when applying human psychometrics to AI systems, pointing to future research directions to establish more appropriate, scenario-aware metrics.

Abstract

As Large Language Models (LLMs) are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.

Identifying Multiple Personalities in Large Language Models with External Evaluation

TL;DR

This paper proposes an external evaluation framework for LLM personality by fine-tuning an MBTI predictor on human MBTI data and applying it to open-ended responses generated by LLMs in two roles (Twitter posts and replies). The authors demonstrate that LLMs exhibit role-dependent personalities, in contrast to humans who show consistent MBTI patterns across roles, highlighting that LLM personality may not be an enduring trait and that human-based personality definitions may not transfer to LLMs. The method combines a three-stage pipeline (model fine-tuning, open-ended response collection, external evaluation with validation on human data) and shows strong per-trait performance (93.3%) with robust overall accuracy (81.0%) for the binary MBTI predictor. The work calls for re-evaluating how personality is defined and measured in LLMs and urges caution when applying human psychometrics to AI systems, pointing to future research directions to establish more appropriate, scenario-aware metrics.

Abstract

As Large Language Models (LLMs) are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.
Paper Structure (18 sections, 5 figures, 8 tables)

This paper contains 18 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personality Theory
  4. LLM Personality Measurement
  5. MBTI Personality Detection Models
  6. Experiments
  7. Stage I: Fine-tuning Llama2-7B-based Personality Detection Model
  8. Stage II: Collecting LLMs Responses Toward Open-ended Situational Questions
  9. Stage III: External LLMs Personality Detection & Validation
  10. Conclusions
  11. Limitation
  12. Acknowledgement
  13. Appendix
  14. Kaggle MBTI Dataset
  15. Computational Resources
  16. ...and 3 more sections

Figures (5)

  • Figure 1: MBTI trait description mbti-fig.
  • Figure 2: Methodology flowchart.
  • Figure 3: MBTI distribution of ChatGPT and Llama2 models using external evaluation method for 100 times. In the figure, the first row is the assessment results on the generated posts dataset (P), whereas the second row provides the results on the comments dataset (C). In addition, personality types that appear less than 3 times are merged together to form the class "Others".
  • Figure 4: MBTI distribution of 4 celebrities using the external evaluation method for 100 times. In the figure, the first row is the assessment results on the generated posts dataset (P), whereas the second row provides the results on the comments dataset (C). In addition, personality types that appear less than 3 times are merged together to form the class "Others". Complete results for all 8 selected celebrities can be found in Appendix \ref{['apd: validation']}.
  • Figure 5: MBTI distribution of 8 selected celebrities using external evaluation method for 100 times. In the figure, the first row is the assessment results on the generated posts dataset (P), whereas the second row provides the results on the comments dataset (C). In addition, personality types that appear less than 3 times are merged together to form the class "Others".