Role-Playing Evaluation for Large Language Models

TL;DR

The paper introduces RPEval, a high-quality, automated, single-turn benchmark for evaluating large language model role-playing across four dimensions: emotional understanding, decision-making, moral alignment, and in-character consistency. It details a data-generation pipeline that uses GPT-4o to produce 3,125 character profiles and 18,850 single-turn scenarios, followed by crowdsourced annotation and majority-vote labeling to yield a robust evaluation set. Empirical results on GPT-4o, Gemini-1.5-Pro, and Llama 3.2 1B show that Gemini-1.5-Pro achieves the highest average performance, while GPT-4o excels in decision-making but struggles with in-character consistency, highlighting model-specific strengths and weaknesses. The work provides a reproducible framework for comparing prompting strategies and model configurations, discusses limitations of single-turn assessments, and outlines future hybrid approaches and safety considerations for responsible use.

Abstract

Large Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing. However, evaluating this ability presents significant challenges, as human assessments are resource-intensive and automated evaluations can be biased. To address this, we introduce Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM role-playing capabilities across four key dimensions: emotional understanding, decision-making, moral alignment, and in-character consistency. This article details the construction of RPEval and presents baseline evaluations. Our code and dataset are available at https://github.com/yelboudouri/RPEval