SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue

TL;DR

SEAD addresses data scarcity in multi-turn service dialogues by decoupling user modeling into a Profile Controller and a User Role-Play Model, enabling fair co-evolution through GRPO. The Profile Controller samples diverse initial user states, while the URM provides realistic responses without controlling outcomes, ensuring authentic adversarial learning. The training loop combines curriculum-driven sampling, multi-turn dialogues, phase-based rewards, and Mistake Analysis to keep training difficulty near a golden ratio of success. Empirically, SEAD outperforms open-source and commercial baselines with a 14B model, achieving higher task completion rates and dialogue efficiency while requiring no annotated dialogue data.

Abstract

Large Language Models have demonstrated remarkable capabilities in open-domain dialogues. However, current methods exhibit suboptimal performance in service dialogues, as they rely on noisy, low-quality human conversation data. This limitation arises from data scarcity and the difficulty of simulating authentic, goal-oriented user behaviors. To address these issues, we propose SEAD (Self-Evolving Agent for Service Dialogue), a framework that enables agents to learn effective strategies without large-scale human annotations. SEAD decouples user modeling into two components: a Profile Controller that generates diverse user states to manage training curriculum, and a User Role-play Model that focuses on realistic role-playing. This design ensures the environment provides adaptive training scenarios rather than acting as an unfair adversary. Experiments demonstrate that SEAD significantly outperforms Open-source Foundation Models and Closed-source Commercial Models, improving task completion rate by 17.6% and dialogue efficiency by 11.1%. Code is available at: https://github.com/Da1yuqin/SEAD.

Figures (4)

• Figure 1: Comparison of training paradigms. A. Static data methods are limited by data quality and may learn human violations. B. Traditional self-evolving creates unfair games where user agents dominate outcomes. Our SEAD achieves balanced co-evolution and realistic interactions through decomposed user modeling, forming a fair adversarial game.
• Figure 2: SEAD Framework Overview. SEAD consists of three components: (1) Profile Generator first creates diverse user profiles, then the (2) User Role-Play Model enacts these users to interact with the (3) Service Agent, training agents to adapt to any user. Finally, these dialogue data reflecting service agent capability returns to the Profile Controller, and initiates the next evolving loop.
• Figure 3: SEAD Co-evolutionary Training Loop. The controller samples initial states (Phase 1), which initialize dialogues producing trajectories (Phase 2), used to train the agent with rewards (Phase 3) and compute completion rates (Phase 4), which feed back to adjust sampling distributions, closing the co-evolutionary loop.
• Figure 4: Case Studies of Challenging Interactions. The User Role-play Model generates heterogeneous personas via clustering, such as the "Rude & Irrational" user (Left) and the "AI-Skeptical" user (Right). The Service Agent demonstrates robustness learned from compound rewards, employing Empathy and Identity Defense strategies to prevent hang-ups and ensure task completion.