Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations

TL;DR

This work tackles the instability of questionnaire-based personality evaluation in LLMs by introducing PVNI, an internal-activation–based method that yields stable, explainable trait measurements. PVNI derives a persona direction from contrastive prompts, anchors a neutral score with a neutral prompt, and interpolates along the persona axis to produce a robust Big Five profile, supported by a linear-theory analysis of persona directions as approximately linear subspaces. The approach is validated across multiple open-source LLMs, demonstrating substantially lower prompt-induced variance than self-report or open-ended elicitation across questionnaire and role-play variants. The findings imply PVNI can provide reliable, interpretable model-characterization for evaluation, alignment, and deployment, with future work aimed at broader trait coverage, reduced judge-dependence, and multilingual generalization.

Abstract

Evaluating personality traits in Large Language Models (LLMs) is key to model interpretation, comparison, and responsible deployment. However, existing questionnaire-based evaluation methods exhibit limited stability and offer little explainability, as their results are highly sensitive to minor variations in prompt phrasing or role-play configurations. To address these limitations, we propose an internal-activation-based approach, termed Persona-Vector Neutrality Interpolation (PVNI), for stable and explainable personality trait evaluation in LLMs. PVNI extracts a persona vector associated with a target personality trait from the model's internal activations using contrastive prompts. It then estimates the corresponding neutral score by interpolating along the persona vector as an anchor axis, enabling an interpretable comparison between the neutral prompt representation and the persona direction. We provide a theoretical analysis of the effectiveness and generalization properties of PVNI. Extensive experiments across diverse LLMs demonstrate that PVNI yields substantially more stable personality trait evaluations than existing methods, even under questionnaire and role-play variants.

Figures (7)

• Figure 1: Comparison of self-assessment, questionnaires, and PVNI. Self-assessment and questionnaires are prompt-sensitive, while PVNI is stable and explainable with internal activations.
• Figure 2: PVNI Pipeline for Prompt-Robust Big Five Trait Measurement. The method extracts a persona direction from positive/negative/neutral prompts, anchors scores on neutral behavior via interpolation and projection, and maps the resulting trait estimates into a stable Big Five subspace.
• Figure 3: Big Five trait radar plots under four evaluation protocols across Qwen-2.5-7B. Shaded bands indicate standard deviation over questionnaire variants.
• Figure 4: Boxplots under questionnaire and role-play variants for Qwen-2.5-7B, Llama-3-8B, and Mistral-7B-v0.1.
• Figure 5: Radar plots of Big Five scores under questionnaire and role-play variants. From left to right: Qwen-2.5-7B, Llama-3-8B, and Mistral-7B-v0.1. Shaded bands indicate $\pm$ one standard deviation across prompt sets.

Theorems & Definitions (7)

• Lemma 4.1: Persona vectors as approximate rank-one amplifiers
• Theorem 4.1: Multi-Persona Composition
• Remark 1
• Theorem 4.2: Persona Negation
• Remark 2
• Theorem 4.3: Out-of-Domain Persona Synthesis
• Remark 3