Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing

Hao Xiang, Tianyi Tang, Yang Su, Bowen Yu, An Yang, Fei Huang, Yichang Zhang, Yaojie Lu, Hongyu Lin, Xianpei Han, Jingren Zhou, Junyang Lin, Le Sun

TL;DR

RMTBench tackles the gap in evaluating LLMs for role-playing by shifting focus from character identity to user intent in multi-turn dialogues. It introduces a bilingual (English–Chinese) benchmark with 80 characters and over 8,000 dialogue rounds, guided by five user-intent scenarios and seven evaluation dimensions. The framework combines automatic LLM-based scoring with human annotation, leverages Claude-generated user utterances, and includes quality-controlled data construction, extra-long dialogues, and an extensive evaluation across 10 models. The results highlight the benefits of multi-turn, user-centric evaluation and reveal language-dependent performance trends, offering a practical, scalable resource for evaluating and improving role-playing capabilities in LLMs.

Abstract

Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a \textbf{character-centric} approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive \textbf{user-centric} bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs. All code and datasets will be released soon. We release the datasets at https://huggingface.co/datasets/xiangh/RMTBENCH.

RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing

TL;DR

RMTBench tackles the gap in evaluating LLMs for role-playing by shifting focus from character identity to user intent in multi-turn dialogues. It introduces a bilingual (English–Chinese) benchmark with 80 characters and over 8,000 dialogue rounds, guided by five user-intent scenarios and seven evaluation dimensions. The framework combines automatic LLM-based scoring with human annotation, leverages Claude-generated user utterances, and includes quality-controlled data construction, extra-long dialogues, and an extensive evaluation across 10 models. The results highlight the benefits of multi-turn, user-centric evaluation and reveal language-dependent performance trends, offering a practical, scalable resource for evaluating and improving role-playing capabilities in LLMs.

Abstract

Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a \textbf{character-centric} approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive \textbf{user-centric} bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs. All code and datasets will be released soon. We release the datasets at https://huggingface.co/datasets/xiangh/RMTBENCH.

Paper Structure

This paper contains 40 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. RMTBench Framework
  4. Senario Design
  5. Character Understanding
  6. Character Maintenance
  7. Implicit User Intentions Response
  8. User Preference Awareness and Reasoning
  9. Sensitive User Behavior Handling
  10. Evaluation Dimension
  11. RMTBench Construction
  12. Character Collection
  13. User-Centric Dialogue Construction
  14. Character Understanding and Character Maintenance
  15. Implicit User Intention Response
  16. ...and 25 more sections

Figures (4)

  • Figure 1: (Left) Character-centric dialogues artificially construct interactions from isolated Q&A pairs, focusing solely on character information while neglecting the necessary cohesive topics and user intentions. (Right) User-centric dialogues prioritize human intentions by structuring conversations around virtual users with clear goals and communicative purposes, creating natural multi-turn interactions that mirror real-world interaction patterns.
  • Figure 2: Construction pipeline of our RMTBench.
  • Figure 3: Comparison results of 3 model responses construction paradigms: multi-turn (used in RMTBench), pseudo multi-turn, and single dialogue block.
  • Figure 4: The average score of each model in each round of the dialogue (has been smoothed).