HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing

TL;DR

HER tackles the challenge of simulating inner reasoning in LLM role-play by introducing Dual-layer Thinking and a three-stage reverse-synthesis pipeline to produce reasoning-augmented trajectories. It trains a context-aware Generative Reward Model (GenRM) distilled from expert principles to provide preference signals and then applies reinforcement learning to tune a Qwen3-32B-based role-play agent. On CoSER, HER achieves a 30.26-point improvement over the Qwen3-32B baseline, and on Minimax Role-Play Bench a 14.97-point gain, while also improving narrative consistency and character fidelity. The work releases datasets, principles, and models to accelerate future research and emphasizes balanced data and pattern diversity to defend against reward hacking.

Abstract

LLM role-playing, i.e., using LLMs to simulate specific personas, has emerged as a key capability in various applications, such as companionship, content creation, and digital games. While current models effectively capture character tones and knowledge, simulating the inner thoughts behind their behaviors remains a challenge. Towards cognitive simulation in LLM role-play, previous efforts mainly suffer from two deficiencies: data with high-quality reasoning traces, and reliable reward signals aligned with human preferences. In this paper, we propose HER, a unified framework for cognitive-level persona simulation. HER introduces dual-layer thinking, which distinguishes characters' first-person thinking from LLMs' third-person thinking. To bridge these gaps, we curate reasoning-augmented role-playing data via reverse engineering and construct human-aligned principles and reward models. Leveraging these resources, we train \method models based on Qwen3-32B via supervised and reinforcement learning. Extensive experiments validate the effectiveness of our approach. Notably, our models significantly outperform the Qwen3-32B baseline, achieving a 30.26 improvement on the CoSER benchmark and a 14.97 gain on the Minimax Role-Play Bench. Our datasets, principles, and models will be released to facilitate future research.

Figures (10)

• Figure 1: The reasoning-driven LLM role-play framework of HERHER introduces Dual-layer Thinking and a three-stage reverse synthesis pipeline to construct reasoning-augmented LLM role-play trajectories
• Figure 2: Overview of HER trainingTop: we train a Role-play GRM by distilling reusable principles from real conversational preference data, and teaching the model to do pairwise judging with by-case principles$\rightarrow$ analysis $\rightarrow$ final decision. Bottom: we first cold-start the LLM role-play model with SFT on HER data, and then apply RL where the GRM compares the policy response with a baseline response to produce the reward.
• Figure 3: Performance of HER Role-play RL training on CoSER Benchmark
• Figure 4: Pattern collapse vs. stable dimension-wise judgments during training
• Figure 5: Effect of system thinking and RL on CoSER Benchmark We compare a base model, SFT without thinking, SFT with system_thinking, and RL model