Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalities at Play: Probing Alignment in AI Teammates

Mohammad Amin Samadi, Nia Nixon

TL;DR

Results suggest that AI personality is measurable but multi-layered and context-dependent, and that evaluating personality-aligned AI teammates requires attention to memory and system-level design, not conversation-only behavior.

Abstract

Collaborative problem solving and learning are shaped by who or what is on the team. As large language models (LLMs) increasingly function as collaborators rather than tools, a key question is whether AI teammates can be aligned to express personality in predictable ways that matter for interaction and learning. We investigate AI personality alignment through a three-lens evaluation framework spanning self-perception (standardized self-report), behavioral expression (team dialogue), and reflective expression (memory construction). We first administered the Big Five Inventory (BFI-44) to LLM-based teammates across four providers (GPT-4o, Claude-3.7 Sonnet, Gemini-2.5 Pro, Grok-3), 32 high/low trait configurations, and multiple prompting strategies. LLMs produced sharply differentiated Big Five profiles, but prompt semantic richness added little beyond simple trait assignment, while provider differences and baseline "default" personalities were substantial. Role framing also mattered: several models refused the assessment without context, yet complied when framed as a collaborative teammate. We then simulated AI participation in authentic team transcripts using high-trait personas and analyzed both generated utterances and structured long-term memories with LIWC-22. Personality signals in conversation were generally subtle and most detectable for Extraversion, whereas memory representations amplified trait-specific signals, especially for Neuroticism, Conscientiousness, and Agreeableness; Openness remained difficult to elicit robustly. Together, results suggest that AI personality is measurable but multi-layered and context-dependent, and that evaluating personality-aligned AI teammates requires attention to memory and system-level design, not conversation-only behavior.

Personalities at Play: Probing Alignment in AI Teammates

TL;DR

Results suggest that AI personality is measurable but multi-layered and context-dependent, and that evaluating personality-aligned AI teammates requires attention to memory and system-level design, not conversation-only behavior.

Abstract

Collaborative problem solving and learning are shaped by who or what is on the team. As large language models (LLMs) increasingly function as collaborators rather than tools, a key question is whether AI teammates can be aligned to express personality in predictable ways that matter for interaction and learning. We investigate AI personality alignment through a three-lens evaluation framework spanning self-perception (standardized self-report), behavioral expression (team dialogue), and reflective expression (memory construction). We first administered the Big Five Inventory (BFI-44) to LLM-based teammates across four providers (GPT-4o, Claude-3.7 Sonnet, Gemini-2.5 Pro, Grok-3), 32 high/low trait configurations, and multiple prompting strategies. LLMs produced sharply differentiated Big Five profiles, but prompt semantic richness added little beyond simple trait assignment, while provider differences and baseline "default" personalities were substantial. Role framing also mattered: several models refused the assessment without context, yet complied when framed as a collaborative teammate. We then simulated AI participation in authentic team transcripts using high-trait personas and analyzed both generated utterances and structured long-term memories with LIWC-22. Personality signals in conversation were generally subtle and most detectable for Extraversion, whereas memory representations amplified trait-specific signals, especially for Neuroticism, Conscientiousness, and Agreeableness; Openness remained difficult to elicit robustly. Together, results suggest that AI personality is measurable but multi-layered and context-dependent, and that evaluating personality-aligned AI teammates requires attention to memory and system-level design, not conversation-only behavior.
Paper Structure (41 sections, 3 figures, 7 tables)

This paper contains 41 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personality and Collaboration
  4. Persona and User Perception
  5. Persona and LLMs
  6. Current Research
  7. Methods
  8. Study 1 - BFI Survey
  9. Persona Generation and Prompting
  10. BFI-44 Administration
  11. Experimental Design and Scoring
  12. Study 2 - Simulation
  13. Simulation Pipeline
  14. Measures
  15. Results -- Study 1
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Multi-lens framework for evaluating AI personality alignment. The AI teammate is assessed through three complementary lenses: (1) Self-Perception via BFI-44 survey responses, (2) Behavioral expression in simulated team conversation, and (3) Reflective memory construction. Outputs are analyzed using LIWC-22 and statistical testing.
  • Figure 2: Simulation pipeline. The agent updates memory (STM + periodic LTM), decides whether to contribute, and either stays silent or generates a persona-grounded response.
  • Figure 3: Effect of Personality Prompting on Big Five Trait Scores. Mean actual trait scores when models were prompted for high (orange) versus low (blue) target levels. Error bars represent standard errors. All differences are significant at $p < .001$.