Contact Complexity in Customer Service

TL;DR

The paper tackles the challenge of routing customer-service contacts by defining a measurable complexity score to direct issues to appropriate agents. It trains an AI expert to mimic senior agents by predicting SIC codes from transcripts using TF-IDF features and LightGBM, and derives a complexity measure from three attributes: transcript length $\mathcal{L}$, uncertainty $\mathcal{E}$, and skillfulness $\mathcal{S}$, combining them into an absolute score $\mathcal{C}$ and a relative score $\mathcal{Q}$ through quantile normalization. The key contributions are a scalable, annotation-free method to compute $\mathcal{C}$ and $\mathcal{Q}$, plus a set of input features and validation through indirect and direct evidence that align with human judgments of complexity. This approach enables complexity-guided routing, potentially reducing transfers and cost, and it is extendable with ensemble AI experts and broader service-labeling to improve robustness and applicability across e-commerce domains.

Abstract

Customers who reach out for customer service support may face a range of issues that vary in complexity. Routing high-complexity contacts to junior agents can lead to multiple transfers or repeated contacts, while directing low-complexity contacts to senior agents can strain their capacity to assist customers who need professional help. To tackle this, a machine learning model that accurately predicts the complexity of customer issues is highly desirable. However, defining the complexity of a contact is a difficult task as it is a highly abstract concept. While consensus-based data annotation by experienced agents is a possible solution, it is time-consuming and costly. To overcome these challenges, we have developed a novel machine learning approach to define contact complexity. Instead of relying on human annotation, we trained an AI expert model to mimic the behavior of agents and evaluate each contact's complexity based on how the AI expert responds. If the AI expert is uncertain or lacks the skills to comprehend the contact transcript, it is considered a high-complexity contact. Our method has proven to be reliable, scalable, and cost-effective based on the collected data.

Figures (7)

• Figure 1: The KL divergence boosting function is demonstrated for two different examples. The boosting model has 60 trees. The divergence boosting function of Example 1 decays faster compared to Example 2, resulting in a smaller integral, or skillfulness, for Example 1. It is important to note that the divergence boosting always reaches zero when the tree index reaches its maximum number.
• Figure .1: Our proposed routing process involves two steps. Firstly, a complexity model will assess the contact to determine whether it should be routed directly to senior or junior agents. If the contact does not meet the criteria for direct routing, a product-line-based model (here we use Amazon's products to demonstrate the idea) will then analyze the intent of the contact and direct it to the appropriate agents.
• Figure 2: (a)-(c) show the distribution of 450K training contacts for different hypotheses: (a) $\mathcal{S}$, (b) $\mathcal{E}$, and (c) $\mathcal{L}$. (Note: To protect Amazon's business from potential information leaks, we rescaled $\mathcal{S}$, $\mathcal{E}$ and $\mathcal{L}$ between 0 and 1. However, this rescaling does not affect the findings presented in this article since only the relative values are significant in the subsequent distribution transformations.) (d)-(f) show the distribution of the 450K training contacts after quantile transformation, which converts all hypotheses to a normal distribution. The transformed hypotheses are referred to as (d) $\mathcal{S}^N$, (e) $\mathcal{E}^N$, (f) $\mathcal{L}^N$, respectively.
• Figure .2: Example of a transcript. The highlighted sentence reflects the topic / customer's issue of this contact.
• Figure 3: (a)-(c) show the distribution of the absolute complexity score for different weights $w$ on $\mathcal{L}^N$. As $w$ increases, the distribution changes from negatively skewed to symmetrical. (d) shows the relative complexity score, obtained by performing a quantile transform on (c) to map it to a uniform distribution.