Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAIDEN-R1: Improving Role-awareness of LLMs via GRPO with Verifiable Reward

Zongsheng Wang, Kaili Sun, Bowen Wu, Qun Yu, Ying Li, Baoxun Wang

TL;DR

This work tackles role-drift in role-playing conversational agents by introducing RAIDEN-R1 with Verifiable Role-Awareness Rewards (VRAR) and a quantifiable reinforcement learning scheme (GRPO). It presents two data-collection pipelines (Single-Term Validation and Multi-Term Dynamic Parsing) and a cold-start SFT workflow to produce a high-quality, role-aware CoT corpus. Experiments on the RAIDEN benchmark demonstrate that 14B-GRPO with VRAR outperforms SFT baselines in key SBK and CM metrics and shows robust reasoning under conflicting cues. The study provides insights into how quantifiable rewards can bridge non-quantifiability gaps in RPCA training and guides future work toward larger models and richer evaluation metrics.

Abstract

Role-playing conversational agents (RPCAs) face persistent challenges in maintaining role consistency. To address this, we propose RAIDEN-R1, a novel reinforcement learning framework that integrates Verifiable Role-Awareness Reward (VRAR). The method introduces both singular and multi-term mining strategies to generate quantifiable rewards by assessing role-specific keys. Additionally, we construct a high-quality, role-aware Chain-of-Thought dataset through multi-LLM collaboration, and implement experiments to enhance reasoning coherence. Experiments on the RAIDEN benchmark demonstrate RAIDEN-R1's superiority: our 14B-GRPO model achieves 88.04% and 88.65% accuracy on Script-Based Knowledge and Conversation Memory metrics, respectively, outperforming baseline models while maintaining robustness. Case analyses further reveal the model's enhanced ability to resolve conflicting contextual cues and sustain first-person narrative consistency. This work bridges the non-quantifiability gap in RPCA training and provides insights into role-aware reasoning patterns, advancing the development of RPCAs.

RAIDEN-R1: Improving Role-awareness of LLMs via GRPO with Verifiable Reward

TL;DR

This work tackles role-drift in role-playing conversational agents by introducing RAIDEN-R1 with Verifiable Role-Awareness Rewards (VRAR) and a quantifiable reinforcement learning scheme (GRPO). It presents two data-collection pipelines (Single-Term Validation and Multi-Term Dynamic Parsing) and a cold-start SFT workflow to produce a high-quality, role-aware CoT corpus. Experiments on the RAIDEN benchmark demonstrate that 14B-GRPO with VRAR outperforms SFT baselines in key SBK and CM metrics and shows robust reasoning under conflicting cues. The study provides insights into how quantifiable rewards can bridge non-quantifiability gaps in RPCA training and guides future work toward larger models and richer evaluation metrics.

Abstract

Role-playing conversational agents (RPCAs) face persistent challenges in maintaining role consistency. To address this, we propose RAIDEN-R1, a novel reinforcement learning framework that integrates Verifiable Role-Awareness Reward (VRAR). The method introduces both singular and multi-term mining strategies to generate quantifiable rewards by assessing role-specific keys. Additionally, we construct a high-quality, role-aware Chain-of-Thought dataset through multi-LLM collaboration, and implement experiments to enhance reasoning coherence. Experiments on the RAIDEN benchmark demonstrate RAIDEN-R1's superiority: our 14B-GRPO model achieves 88.04% and 88.65% accuracy on Script-Based Knowledge and Conversation Memory metrics, respectively, outperforming baseline models while maintaining robustness. Case analyses further reveal the model's enhanced ability to resolve conflicting contextual cues and sustain first-person narrative consistency. This work bridges the non-quantifiability gap in RPCA training and provides insights into role-aware reasoning patterns, advancing the development of RPCAs.
Paper Structure (14 sections, 5 figures, 1 table, 2 algorithms)

This paper contains 14 sections, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Reinforcement Learning
  4. Data Collecting
  5. Single-Term Validation
  6. Multi-Term Dynamic Parsing
  7. Reward Design
  8. Cold-start SFT
  9. Experiments
  10. Dataset and Experiment Setup
  11. Evaluation Method
  12. Results
  13. Case Analysis
  14. Next Steps

Figures (5)

  • Figure 1: Workflow of VRAR, followed by the multi-stage training strategy of DeepSeek R1, we create two datasets for sft and reinforcement learning.
  • Figure 2: Accuracy Reward
  • Figure 3: An example of the generated cold-start CoT dataset
  • Figure 4: Camparisions between responses from 14B-Instruct and 14B-GRPO under different query objectives
  • Figure :