Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Generalization in Role-Playing Models via Information Theory

Yongqi Li, Hao Lang, Fei Huang, Tieyun Qian, Yongbin Li

TL;DR

This work analyzes generalization gaps in role-playing models (RPMs) under distribution shifts across user, character, and dialogue contexts. It introduces R-EMID, an information-theoretic metric built on reasoning-augmented mutual information to quantify RPM degradation and derives an upper bound to predict worst-case performance. A co-evolving reinforcement learning (CoRL) framework is proposed to estimate R-EMID by jointly learning a reasoning generator and a conditional response model, enabling robust probability estimation. Empirical results using RPGBench show that user shifts pose the greatest risk, RL-based methods yield consistent generalization gains, and naive reasoning traces do not improve performance, validating the proposed framework and highlighting practical guidance for RPM deployment.

Abstract

Role-playing models (RPMs) are widely used in real-world applications but underperform when deployed in the wild. This degradation can be attributed to distribution shifts, including user, character, and dialogue compositional shifts. Existing methods like LLM-as-a-judge fall short in providing a fine-grained diagnosis of how these shifts affect RPM generalization, and thus there lack formal frameworks to characterize RPM generalization behaviors. To bridge these gaps, we introduce an information-theoretic metric, named reasoning-based effective mutual information difference (R-EMID), to measure RPM performance degradation in an interpretable way. We also derive an upper bound on R-EMID to predict the worst-case generalization performance of RPMs and theoretically reveal how various shifts contribute to the RPM performance degradation. Moreover, we propose a co-evolving reinforcement learning framework to adaptively model the connection among user, character, and dialogue context and thus enhance the estimation of dialogue response generation probability, which is critical for calculating R-EMID. Finally, we evaluate the generalization performance of various RPMs using R-EMID, finding that user shift poses the highest risk among all shifts and reinforcement learning is the most effective approach for enhancing RPM generalization.

Understanding Generalization in Role-Playing Models via Information Theory

TL;DR

This work analyzes generalization gaps in role-playing models (RPMs) under distribution shifts across user, character, and dialogue contexts. It introduces R-EMID, an information-theoretic metric built on reasoning-augmented mutual information to quantify RPM degradation and derives an upper bound to predict worst-case performance. A co-evolving reinforcement learning (CoRL) framework is proposed to estimate R-EMID by jointly learning a reasoning generator and a conditional response model, enabling robust probability estimation. Empirical results using RPGBench show that user shifts pose the greatest risk, RL-based methods yield consistent generalization gains, and naive reasoning traces do not improve performance, validating the proposed framework and highlighting practical guidance for RPM deployment.

Abstract

Role-playing models (RPMs) are widely used in real-world applications but underperform when deployed in the wild. This degradation can be attributed to distribution shifts, including user, character, and dialogue compositional shifts. Existing methods like LLM-as-a-judge fall short in providing a fine-grained diagnosis of how these shifts affect RPM generalization, and thus there lack formal frameworks to characterize RPM generalization behaviors. To bridge these gaps, we introduce an information-theoretic metric, named reasoning-based effective mutual information difference (R-EMID), to measure RPM performance degradation in an interpretable way. We also derive an upper bound on R-EMID to predict the worst-case generalization performance of RPMs and theoretically reveal how various shifts contribute to the RPM performance degradation. Moreover, we propose a co-evolving reinforcement learning framework to adaptively model the connection among user, character, and dialogue context and thus enhance the estimation of dialogue response generation probability, which is critical for calculating R-EMID. Finally, we evaluate the generalization performance of various RPMs using R-EMID, finding that user shift poses the highest risk among all shifts and reinforcement learning is the most effective approach for enhancing RPM generalization.

Paper Structure

This paper contains 46 sections, 3 theorems, 18 equations, 11 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Formulation
  4. Distribution Shifts in RPMs
  5. Pilot Study
  6. Understanding RPM Generalization via Information Theory
  7. Preliminary
  8. Reasoning-based EMID
  9. R-EMID Estimation
  10. R-EMID Validation
  11. Effectiveness of R-EMID
  12. Analysis on R-EMID Upper Bound
  13. Analysis on CoRL
  14. Evaluating RPMs using R-EMID
  15. Experimental Setup
  16. ...and 31 more sections

Key Result

Theorem 3.4

Given an RPM $P_{\theta}$ trained on the distribution $P_{XY}$ and tested on $Q_{XY}$, which share consistent conditional distributions $Y|X$. If there exist constants $\delta_{P}$ and $\delta_{Q}$ such that $D_{\rm JS}(P_{Y_{{\theta}}}\|P_{Y})\leq \delta_{P}, D_{\rm JS}(Q_{Y_{{\theta}}}\|Q_{Y})\leq where $\widehat{H}=\max_{x \in \mathcal{X}} [H(Q_{Y|x,r})+H(P_{Y_{\theta}|x,r})]$, and $r=f_R(x)$ d

Figures (11)

  • Figure 1: Distribution shifts in RPMs. (a) User shift: distribution change in user persona, e.g., EN$\rightarrow$ZH; (b) Character shift: distribution change in agent character, e.g., unseen character types; (c) Dialogue compositional shift: distribution change in dialogues, e.g., from short-turn dialogue to composed long-turn dialogue.
  • Figure 2: Pilot study of RPM performance under various distribution shifts, including user shift (left), character shift (middle), and dialogue compositional shift (right). The RPMs are trained with supervised fine-tuning based on six LLMs (extended results based on more LLMs are provided in Appendix \ref{['sec:app:pilot_study_more_results']}). Here, the x-axis is sorted by the severity of the distribution shifts, while the y-axis represents RPM performance measured by WinRate. We observe a consistent trend: as the severity of the distribution shift increases, performance degrades more significantly.
  • Figure 3: The proposed CoRL method for estimating R-EMID, which includes two modules for reasoning generation and conditional probability estimation, respectively. The two modules are optimized alternately during the co-evolution.
  • Figure 4: Scatter plots with regression lines showing the correlations between information-theoretic metrics and LLM-as-a-judge metrics: (a) EMI vs. WinRate Score; (b) EMID vs. WinRate Score Difference; (c) R-EMI (Ours) vs. WinRate Score; (d) R-EMID (Ours) vs. WinRate Score Difference. The legend reports the correlation coefficients along with $p$-values.
  • Figure 5: (a) Trend plot showing the correlation between the estimated R-EMID upper bound and R-EMID when the number of samples used for bound estimation increases. (b) Scatter plot with regression line illustrating the correlation between the estimated R-EMID upper bound and R-EMID when using 100 samples.
  • ...and 6 more figures

Theorems & Definitions (7)

  • Definition 3.1: EMID
  • Definition 3.2: Reasoning-based EMI, R-EMI
  • Definition 3.3: Reasoning-based EMID, R-EMID
  • Theorem 3.4: Upper Bound on R-EMID
  • Theorem C.1: Upper Bound on R-EMID
  • proof : Proof of Theorem \ref{['theorem:restatement:REMID_upperbound']}
  • Theorem C.2: Restatement of Upper Bound on EMID oh-2025-understanding