Evaluating, Synthesizing, and Enhancing for Customer Support Conversation

TL;DR

This work addresses the challenge of training customer support agents to respond with structured, empathetic strategies by introducing the CSC framework grounded in COPC guidelines. It constructs CSConv, a high-quality dataset of real dialogues rewritten to reflect deliberate strategy use, and RoleCS, a large synthetic dataset generated via multi-role LLM-based simulations. Fine-tuning strong models on RoleCS significantly improves strategy-aligned response generation on CSConv, with human evaluations confirming better problem resolution and interaction quality. The combination of strategy-guided data and role-playing synthetic data offers a scalable path to more effective, empathetic customer support systems, and the authors publicly release code and data.

Abstract

Effective customer support requires not only accurate problem solving but also structured and empathetic communication aligned with professional standards. However, existing dialogue datasets often lack strategic guidance, and real-world service data is difficult to access and annotate. To address this, we introduce the task of Customer Support Conversation (CSC), aimed at training customer service agents to respond using well-defined support strategies. We propose a structured CSC framework grounded in COPC guidelines, defining five conversational stages and twelve strategies to guide high-quality interactions. Based on this, we construct CSConv, an evaluation dataset of 1,855 real-world customer-agent conversations rewritten using LLMs to reflect deliberate strategy use, and annotated accordingly. Additionally, we develop a role-playing approach that simulates strategy-rich conversations using LLM-powered roles aligned with the CSC framework, resulting in the training dataset RoleCS. Experiments show that fine-tuning strong LLMs on RoleCS significantly improves their ability to generate high-quality, strategy-aligned responses on CSConv. Human evaluations further confirm gains in problem resolution. All code and data will be made publicly available at https://github.com/aliyun/qwen-dianjin.

Figures (30)

• Figure 1: An example dialogue between a service supporter (left) and a customer (right), showing support strategies (noted in parentheses) used by the supporter. The conversation is organized into five stages of the proposed CSC framework, shown in colored boxes.
• Figure 2: Overview of the CSC framework's five stages, each paired with recommended support strategies (see Table \ref{['tbl:strategy']} in Appendix \ref{['apdx:csc_framework']}). The typical flow is: ① Connecting → ② Identifying → ③ Exploring → ④ Resolving → ⑤ Maintaining (black arrows), but it can be adjusted based on the specifics of each conversation (dashed arrows).
• Figure 3: Strategy proportion of CSConv.
• Figure 4: Illustration of synthetic conversation generation using role-playing agents.
• Figure 5: Word overlap between cumulative customer utterances and profile (Aligned vs. Random).