Persona Alchemy: Designing, Evaluating, and Implementing Psychologically-Grounded LLM Agents for Diverse Stakeholder Representation
Sola Kim, Dongjune Chang, Jieshu Wang
TL;DR
This work addresses the challenge of creating psychologically grounded LLM agents that represent diverse stakeholders in interdisciplinary contexts. It introduces a Social Cognitive Theory–based framework that operationalizes four personal factors and six SCT constructs within a Neo4j graph- backed architecture, leveraging Retrieval-Augmented Generation and memory to maintain longitudinal persona fidelity. Across a simulated renewable energy case study with five diverse agents, the approach yields consistent response patterns (R^2 in the range $[0.58,0.61]$) and reveals a two-component PCA explaining $73\%$ of SCT-construct variance, with clear temporal dynamics—self-efficacy increases over rounds while certain other constructs decline. The framework offers improved explainability and reproducibility over static prompts, enabling more authentic stakeholder representation and long-term persona development in AI systems, while acknowledging domain-generalizability and ethical considerations for broader deployment.
Abstract
Despite advances in designing personas for Large Language Models (LLM), challenges remain in aligning them with human cognitive processes and representing diverse stakeholder perspectives. We introduce a Social Cognitive Theory (SCT) agent design framework for designing, evaluating, and implementing psychologically grounded LLMs with consistent behavior. Our framework operationalizes SCT through four personal factors (cognitive, motivational, biological, and affective) for designing, six quantifiable constructs for evaluating, and a graph database-backed architecture for implementing stakeholder personas. Experiments tested agents' responses to contradicting information of varying reliability. In the highly polarized renewable energy transition discourse, we design five diverse agents with distinct ideologies, roles, and stakes to examine stakeholder representation. The evaluation of these agents in contradictory scenarios occurs through comprehensive processes that implement the SCT. Results show consistent response patterns ($R^2$ range: $0.58-0.61$) and systematic temporal development of SCT construct effects. Principal component analysis identifies two dimensions explaining $73$% of variance, validating the theoretical structure. Our framework offers improved explainability and reproducibility compared to black-box approaches. This work contributes to ongoing efforts to improve diverse stakeholder representation while maintaining psychological consistency in LLM personas.