MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents

TL;DR

The paper introduces Multimodal Role-Playing Agents (MRPAs) to extend character-based interactions from text to vision-language modalities. It presents MMRole, a framework consisting of MMRole-Data for large-scale, high-quality multimodal datasets and MMRole-Eval for robust ground-truth-based evaluation, plus a specialized MRPA, MMRole-Agent. MMRole-Data comprises 85 characters, about 11K images, and 14K dialogues across three dialogue types, while MMRole-Eval scores MRPAs with eight metrics via a reward-model approach trained to reflect GPT-4 and human judgments. Experiments show MMRole-Agent achieves strong performance and generalization, highlighting the importance of data quality and diverse character representation for multimodal role-playing, and identifying remaining challenges in multimodal understanding and role-consistency.

Abstract

Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research. However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities. To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation approach. Specifically, we construct a large-scale, high-quality dataset, MMRole-Data, consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues. Additionally, we present a robust evaluation approach, MMRole-Eval, encompassing eight metrics across three dimensions, where a reward model is designed to score MRPAs with the constructed ground-truth data for comparison. Moreover, we develop the first specialized MRPA, MMRole-Agent. Extensive evaluation results demonstrate the improved performance of MMRole-Agent and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understanding and role-playing consistency. The data, code, and models are all available at https://github.com/YanqiDai/MMRole.

Figures (19)

• Figure 1: An overview of the MMRole framework. (a) MMRole-Data includes character profiles, images, and dialogues centered around images. (b) MMRole-Eval comprises eight evaluation metrics across three dimensions. For each metric, the reward model scores MRPAs with the constructed ground-truth data for comparison.
• Figure 2: Examples of the three types of dialogue scenarios in MMRole-Data.
• Figure 3: The visualization of the evaluation results for all MRPAs. Each indicator displays an interval length of $0.55$, and the maximum value of the interval for different indicators is adjusted from $1.10$ to $1.25$.
• Figure 4: All character constructed in MMRole-Data, with the series to which the characters in the out-of-distribution test set belong being underlined.
• Figure 5: The character profile of Iron Man from The Avengers.