Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

TL;DR

This work establishes a fairer evaluation protocol and validates a scalable, personality-enhanced framework for constructing robust RPAs, and demonstrates that incorporating personality information consistently improves RPA performance.

Abstract

Large language models (LLMs) have demonstrated significant potential in developing Role-Playing Agents (RPAs). However, current research primarily evaluates RPAs using famous fictional characters, allowing models to rely on memory associated with character names. This dependency creates a bias that limits the generalization of RPAs to unseen personas. To address this issue, we propose an anonymous evaluation method. Experiments across multiple benchmarks reveal that anonymization significantly degrades role-playing performance, confirming that name exposure carries implicit information. Furthermore, we investigate personality augmentation to enhance role fidelity under anonymous setting. We systematically compare the efficacy of personality traits derived from human annotations versus those self-generated by the model. Our results demonstrate that incorporating personality information consistently improves RPA performance. Crucially, self-generated personalities achieve performance comparable to human-annotated ones. This work establishes a fairer evaluation protocol and validates a scalable, personality-enhanced framework for constructing robust RPAs.

Figures (6)

• Figure 1: The framework of constructing a personality augmented role-playing agent under anonymous scenarios.
• Figure 2: Paired response evaluation on RoleAgentBench: win rates of original vs. anonymous settings.
• Figure 3: Pairwise comparison on RoleAgentBench: MBTI or Big Five personality integration vs. the original condition.
• Figure 4: Human pairwise comparison against the original condition.
• Figure 5: Prompt template for personality-augmented RPA.