Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning

Feng Zhang, Shijia Li, Chunmao Zhang, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Jingwen Xu, Han Liu

TL;DR

The paper tackles the lack of generalizable, strategically capable user simulators by introducing UserLM-R1, a reasoning-enabled simulator built on generalizable static and dynamic profiles. It combines a goal-driven reasoning policy that generates explicit reasoning traces with supervised fine-tuning and multi-reward reinforcement learning to cultivate proactive, manipulation-resistant behavior. Extensive evaluation, including a challenging adversarial dataset, shows that UserLM-R1 achieves superior session- and turn-level performance and enhances RL agent training in real-world tasks. The work advances user simulation by modeling human-like decision processes and dynamic goals, offering practical benefits for scalable agent development and robust evaluation in multi-domain settings.

Abstract

User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set.

UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning

TL;DR

The paper tackles the lack of generalizable, strategically capable user simulators by introducing UserLM-R1, a reasoning-enabled simulator built on generalizable static and dynamic profiles. It combines a goal-driven reasoning policy that generates explicit reasoning traces with supervised fine-tuning and multi-reward reinforcement learning to cultivate proactive, manipulation-resistant behavior. Extensive evaluation, including a challenging adversarial dataset, shows that UserLM-R1 achieves superior session- and turn-level performance and enhances RL agent training in real-world tasks. The work advances user simulation by modeling human-like decision processes and dynamic goals, offering practical benefits for scalable agent development and robust evaluation in multi-domain settings.

Abstract

User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set.
Paper Structure (32 sections, 3 equations, 16 figures, 4 tables)

This paper contains 32 sections, 3 equations, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. User Simulation
  4. Reinforcement Learning for Reasoning
  5. User Simulator Evaluation
  6. Task Formulation
  7. UserLM with Reasoning
  8. General User Profile Construction
  9. Goal-Driven Decision-Making Strategy
  10. Bootstrapping via Supervised Fine-Tuning
  11. Generalizing with Multi-Reward Reinforcement Learning
  12. Experiments
  13. Experimental Setup
  14. Evaluation Metrics
  15. Main Results
  16. ...and 17 more sections

Figures (16)

  • Figure 1: Comparison of previous methods and ours.
  • Figure 2: The overview of our proposed UserLM-R1. First, general user profiles are constructed. Second, the goal-driven decision-making strategy enables the user model to have reasoning capability. Finally, supervised fine-tuning and multi-reward reinforcement learning are used to further improve the proactive strategic ability.
  • Figure 3: Quality and expression style analysis of the constructed user profiles.
  • Figure 4: Evaluation results of retail and hiring agents trained with DeepSeek-R1 and UserLM-R1.
  • Figure 5: Semantic distribution of multi-dimensional attributes within the persona embedding space.
  • ...and 11 more figures