Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLM Agents Maintain a Persona in Discourse?

Pranav Bhandari, Nicolas Fay, Michael Wise, Amitava Datta, Stephanie Meek, Usman Naseem, Mehwish Nasim

TL;DR

This work addresses whether LLM agents can maintain assigned Big Five (OCEAN) personality traits during dyadic discourse. It introduces an agent-based framework with two trait-annotated agents and judge agents that infer and evaluate trait adherence using metrics such as HTA/LTA, Fleiss' kappa, and LIWC-22 linguistic markers. Results show that while trait-guided dialogue is possible, its consistency is highly dependent on model pairing, topic, and interlocutor, with notable variability across Extraversion and Openness. The study highlights the need for standardized benchmarks and improved evaluation methods to achieve stable, interpretable personality-driven interactions in LLMs, pointing to future work in prompting, fine-tuning, and reinforcement learning for more robust trait maintenance.

Abstract

Large Language Models (LLMs) are widely used as conversational agents, exploiting their capabilities in various sectors such as education, law, medicine, and more. However, LLMs are often subjected to context-shifting behaviour, resulting in a lack of consistent and interpretable personality-aligned interactions. Adherence to psychological traits lacks comprehensive analysis, especially in the case of dyadic (pairwise) conversations. We examine this challenge from two viewpoints, initially using two conversation agents to generate a discourse on a certain topic with an assigned personality from the OCEAN framework (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) as High/Low for each trait. This is followed by using multiple judge agents to infer the original traits assigned to explore prediction consistency, inter-model agreement, and alignment with the assigned personality. Our findings indicate that while LLMs can be guided toward personality-driven dialogue, their ability to maintain personality traits varies significantly depending on the combination of models and discourse settings. These inconsistencies emphasise the challenges in achieving stable and interpretable personality-aligned interactions in LLMs.

Can LLM Agents Maintain a Persona in Discourse?

TL;DR

This work addresses whether LLM agents can maintain assigned Big Five (OCEAN) personality traits during dyadic discourse. It introduces an agent-based framework with two trait-annotated agents and judge agents that infer and evaluate trait adherence using metrics such as HTA/LTA, Fleiss' kappa, and LIWC-22 linguistic markers. Results show that while trait-guided dialogue is possible, its consistency is highly dependent on model pairing, topic, and interlocutor, with notable variability across Extraversion and Openness. The study highlights the need for standardized benchmarks and improved evaluation methods to achieve stable, interpretable personality-driven interactions in LLMs, pointing to future work in prompting, fine-tuning, and reinforcement learning for more robust trait maintenance.

Abstract

Large Language Models (LLMs) are widely used as conversational agents, exploiting their capabilities in various sectors such as education, law, medicine, and more. However, LLMs are often subjected to context-shifting behaviour, resulting in a lack of consistent and interpretable personality-aligned interactions. Adherence to psychological traits lacks comprehensive analysis, especially in the case of dyadic (pairwise) conversations. We examine this challenge from two viewpoints, initially using two conversation agents to generate a discourse on a certain topic with an assigned personality from the OCEAN framework (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) as High/Low for each trait. This is followed by using multiple judge agents to infer the original traits assigned to explore prediction consistency, inter-model agreement, and alignment with the assigned personality. Our findings indicate that while LLMs can be guided toward personality-driven dialogue, their ability to maintain personality traits varies significantly depending on the combination of models and discourse settings. These inconsistencies emphasise the challenges in achieving stable and interpretable personality-aligned interactions in LLMs.

Paper Structure

This paper contains 21 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Prompt formation
  5. Validation
  6. Evaluation
  7. Personality prediction consistency Across Models:
  8. Inter-rater reliability among the models:
  9. Discourse alignment with Assigned Personality Traits:
  10. Results
  11. Personality Prediction Consistency across models
  12. Inter Model Agreement
  13. Discourse Alignment with assigned personality traits
  14. Conclusion
  15. Limitations
  16. ...and 6 more sections

Figures (3)

  • Figure 1: An example of inducing personality in LLM agents, followed by a discourse. A judge agent evaluates whether personality traits were adhered to in the discourse.
  • Figure 2: Methodology of the paper. System prompt inducing traits and topic of discourse are passed with the User prompt containing previous utterance. The conversations are then extracted and analysed by Judge Agents to report the findings.
  • Figure 3: LIWC analysis depicting the accuracy of conveying the assigned personality traits to Participants 1 and 2.