MOA: Multi-Objective Alignment for Role-Playing Agents

TL;DR

The paper tackles the challenge of training role-playing agents to satisfy multiple, often conflicting rubrics such as instruction following, persona consistency, and knowledge. It introduces MOA, a multi-objective reinforcement learning framework that dynamically selects a pivot dimension, eliminates conflicting rollout samples, and employs thought-augmented rollout with off-policy guidance to balance diverse objectives. Through extensive experiments on PersonaGym and RoleMRC, MOA with an 8B model matches or surpasses strong baselines like GPT-4o and Claude on key metrics, including a 21% average improvement on RoleMRC. These results demonstrate MOA's robustness across model sizes and its potential as a scalable approach to building general RPAs capable of multi-turn reasoning and persona-driven dialogue.

Abstract

Role-playing agents (RPAs) must simultaneously master many conflicting skills -- following multi-turn instructions, exhibiting domain knowledge, and adopting a consistent linguistic style. Existing work either relies on supervised fine-tuning (SFT) that over-fits surface cues and yields low diversity, or applies reinforcement learning (RL) that fails to learn multiple dimensions for comprehensive RPA optimization. We present MOA (Multi-Objective Alignment), a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs. MOA introduces a novel multi-objective optimization strategy that trains simultaneously on multiple fine-grained rubrics to boost optimization performance. Besides, to address the issues of model output diversity and quality, we have also employed thought-augmented rollout with off-policy guidance. Extensive experiments on challenging benchmarks such as PersonaGym and RoleMRC show that MOA enables an 8B model to match or even outperform strong baselines such as GPT-4o and Claude across numerous dimensions. This demonstrates the great potential of MOA in building RPAs that can simultaneously meet the demands of role knowledge, persona style, diverse scenarios, and complex multi-turn conversations.

Figures (7)

• Figure 1: Flowchart of MOA. Given the input $\mathbf{q}$, we first prompt the policy model to generate rollouts with thoughts, and then mix them with off-policy samples. We then score these rollouts using fine-grained rubrics. Based on the reward trends from these rubrics, we dynamically select a pivot dimension for optimization and allocate weights. Finally, we eliminate conflicting samples that hinder optimization in the pivot dimension.
• Figure 2: Prompt $P_{\text{think}}$ to guide models in role-playing tasks.
• Figure 3: Performance of Claude-3.7 on PersonaGym, with and without thinking
• Figure 4: Smoothed training reward curves of Qwen3-8B-SFT across different dimensions. It can be seen that after using multi-objective optimization, the reward always rises faster. Besides, the starting point of MOA-o is higher. This is because introducing thinking leads to a decline in generation quality. However, the growth of the MOA-o curve slows down in the later stages of training.
• Figure 5: Reward prompt for Basic Dialogue

Theorems & Definitions (6)

• Theorem 1
• proof : Proof sketch.
• Theorem 2: Small-$\beta$ positive-improvement bound
• proof
• Corollary 1: Model with additive zero-mean noise
• proof