Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Zhenyu Wang, Xiaofen Xing, Yirong Chen, Xiangmin Xu

TL;DR

MERRY is proposed, a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents that transforms the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations.

Abstract

Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions. However, existing studies still rely on pure textual benchmarks to evaluate the text responses of MRPAs, while delegating the assessment of their multimodal expressions solely to modality-synthesis metrics. This evaluation paradigm, on the one hand, entangles semantic assessment with modality generation, leading to ambiguous error attribution, and on the other hand remains constrained by the heavy reliance on human judgment. To this end, we propose MERRY, a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents. This framework introduce five refined metrics for EC and three for RC. Notably, we transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations. Based on MERRY, we conduct extensive evaluations. Our empirical results primarily reveal that: (1) Training on synthetic datasets tends to reduce emotional consistency, whereas training on real-world datasets improves it; (2) Existing models suffer from emotional templatization and simplification, exhibiting positive-bias and performance bottleneck in fine-grained negative emotions; (3) Simple prompting method strengthens the weak models but constrains the strong ones, while simple fine-tuning method suffers from poor role generalization. Codes and dataset are available.

MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

TL;DR

MERRY is proposed, a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents that transforms the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations.

Abstract

Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions. However, existing studies still rely on pure textual benchmarks to evaluate the text responses of MRPAs, while delegating the assessment of their multimodal expressions solely to modality-synthesis metrics. This evaluation paradigm, on the one hand, entangles semantic assessment with modality generation, leading to ambiguous error attribution, and on the other hand remains constrained by the heavy reliance on human judgment. To this end, we propose MERRY, a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents. This framework introduce five refined metrics for EC and three for RC. Notably, we transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations. Based on MERRY, we conduct extensive evaluations. Our empirical results primarily reveal that: (1) Training on synthetic datasets tends to reduce emotional consistency, whereas training on real-world datasets improves it; (2) Existing models suffer from emotional templatization and simplification, exhibiting positive-bias and performance bottleneck in fine-grained negative emotions; (3) Simple prompting method strengthens the weak models but constrains the strong ones, while simple fine-tuning method suffers from poor role generalization. Codes and dataset are available.
Paper Structure (22 sections, 8 equations, 6 figures, 11 tables)

This paper contains 22 sections, 8 equations, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation of Role-Playing Agents
  4. Multimodal Role-Playing Datasets
  5. MERRY
  6. Preliminaries
  7. Evaluation Task Definition
  8. Emotion Transition in Dialogue
  9. Evaluation Pipeline
  10. Evaluation Metrics
  11. Emotional Consistency (EC)
  12. Role Consistency (RC)
  13. Construction of MERRY-Data
  14. Experiments
  15. Experimental Setup
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Current end-to-end evaluations lead to ambiguous error attribution and strong dependence on human assessment. For instance, given a same video output and end-to-end human evaluation result, the issue can be ambiguously attributed to either MLLM (top) or Talker (bottom).
  • Figure 2: Framework of MERRY. Upper Left: dialogue format of evaluation tasks. Lower Left: evaluation pipeline. Right: eight metrics for emotional and role consistencies.
  • Figure 3: MERRY-Data Construction Pipeline
  • Figure 4: Emotional transition matrices of Andy in Ode to Joy. Groundtruth matrices are list in the first column. The rest columns are role-played by GPT-5-chat with different types.
  • Figure 5: The visualization of Recall and Precision scores of each emotions when calculating $MEC_{lower}$ on all models with $type=All$.
  • ...and 1 more figures