Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue

Ning Gao, Wei Zhang, Yuqin Dai, Ling Shi, Ziyin Wang, Yujie Wang, Wei He, Jinpeng Wang, Chaozheng Wang

TL;DR

This work proposes InteractCS-RL, a framework that reframes task-oriented dialogue as a multi-granularity reinforcement learning process, and establishes a User-centric Interaction Framework to provide a high-fidelity training gym, enabling agents to dynamically explore diverse strategies with persona-driven users.

Abstract

The rapid evolution of Large Language Models (LLMs) has accelerated the transition from conversational chatbots to general agents. However, effectively balancing empathetic communication with budget-aware decision-making remains an open challenge. Since existing methods fail to capture these complex strategic trade-offs, we propose InteractCS-RL, a framework that reframes task-oriented dialogue as a multi-granularity reinforcement learning process. Specifically, we first establish a User-centric Interaction Framework to provide a high-fidelity training gym, enabling agents to dynamically explore diverse strategies with persona-driven users. Then, we introduce Cost-aware Multi-turn Policy Optimization (CMPO) with a hybrid advantage estimation strategy. By integrating generative process credits and employing a PID-Lagrangian cost controller, CMPO effectively guides the policy to explore Pareto boundary between user reward and global cost constraints. Extensive experiments on customized real business scenarios demonstrate that InteractCS-RL significantly outperform other baselines across three evaluation dimensions. Further evaluation on tool-agent-user interaction benchmarks verify InteractCS-RL robustness across diverse domains.

Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue

TL;DR

This work proposes InteractCS-RL, a framework that reframes task-oriented dialogue as a multi-granularity reinforcement learning process, and establishes a User-centric Interaction Framework to provide a high-fidelity training gym, enabling agents to dynamically explore diverse strategies with persona-driven users.

Abstract

The rapid evolution of Large Language Models (LLMs) has accelerated the transition from conversational chatbots to general agents. However, effectively balancing empathetic communication with budget-aware decision-making remains an open challenge. Since existing methods fail to capture these complex strategic trade-offs, we propose InteractCS-RL, a framework that reframes task-oriented dialogue as a multi-granularity reinforcement learning process. Specifically, we first establish a User-centric Interaction Framework to provide a high-fidelity training gym, enabling agents to dynamically explore diverse strategies with persona-driven users. Then, we introduce Cost-aware Multi-turn Policy Optimization (CMPO) with a hybrid advantage estimation strategy. By integrating generative process credits and employing a PID-Lagrangian cost controller, CMPO effectively guides the policy to explore Pareto boundary between user reward and global cost constraints. Extensive experiments on customized real business scenarios demonstrate that InteractCS-RL significantly outperform other baselines across three evaluation dimensions. Further evaluation on tool-agent-user interaction benchmarks verify InteractCS-RL robustness across diverse domains.
Paper Structure (81 sections, 12 equations, 3 figures, 6 tables)

This paper contains 81 sections, 12 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Method
  5. User-Centric Interaction Framework
  6. Persona Profiles Bank
  7. User Simulation with Role-play
  8. Multi-turn Dynamic Interaction
  9. Cost-aware Multi-turn Policy Optimization
  10. Session-level Outcome Utility
  11. Process Credit Assignment
  12. Global Cost Constraints
  13. Experimental Setup
  14. Evaluation Objectives
  15. Benchmarks and Evaluation Metrics
  16. ...and 66 more sections

Figures (3)

  • Figure 1: Illustration of our InteractCS-RL. (a) User-Centric Interaction Framework: Integrates persona bank modeling with dynamic user role-play to generate diverse interactive trajectories.(b) Cost-aware Multi-turn Policy Optimization: Synthesizes session-level outcomes, turn-level generative process credits, and PID-regulated global cost constraints into a hybrid advantage for stable policy optimization.
  • Figure 2: Results of different aspects of ablation studies.
  • Figure 3: Visualization of training dynamics under different cost constraints. The left panel shows the evolution of the Voucher Cost (penalty magnitude), while the right panel shows the corresponding Voucher Rate. The Red line (Target: 20%) exhibits a second penalty spike around step 70, illustrating how the PID controller dynamically increases the cost to suppress the agent's attempt to violate the tighter constraint, whereas the Blue line (Target: 30%) stabilizes more smoothly.