Universal Model in Online Customer Service

TL;DR

The paper tackles the retraining burden in online customer service by introducing a universal, training-free retrieval-based approach. It builds a question-label repository using a SeaCat-based sentence tagging model that extracts each transcript’s main customer question, then uses a BM25-style retrieval to predict both categorical (e.g., product/service) and continuous (e.g., handling time) labels from the Top-$N$ similar questions. The approach is evaluated against a Random Forest baseline and shown to improve with larger repositories, outperforming supervised methods in several settings. This method enables rapid deployment and adaptation to new products or events, reducing model development time and operational costs while maintaining predictive performance.

Abstract

Building machine learning models can be a time-consuming process that often takes several months to implement in typical business scenarios. To ensure consistent model performance and account for variations in data distribution, regular retraining is necessary. This paper introduces a solution for improving online customer service in e-commerce by presenting a universal model for predict-ing labels based on customer questions, without requiring training. Our novel approach involves using machine learning techniques to tag customer questions in transcripts and create a repository of questions and corresponding labels. When a customer requests assistance, an information retrieval model searches the repository for similar questions, and statistical analysis is used to predict the corresponding label. By eliminating the need for individual model training and maintenance, our approach reduces both the model development cycle and costs. The repository only requires periodic updating to maintain accuracy.

Figures (4)

• Figure 1: The Universal Model workflow has two stages. In Stage 1, customer questions are extracted from transcripts using the SeaCat Model, and their attributes are combined to form a repository. In Stage 2, the process begins with the customer entering their question or request into a retrieval model. The retrieval model then searches for the Top-N most similar questions and retrieves their attributes from the repository. Next, the distributions of the retrieved attributes are analyzed to make predictions. Categorical labels are predicted using normalized frequency (probability), while continuous labels are predicted using their median.
• Figure 1: An illustration of the functioning of the SeaCat model (derived from a synthetic research dataset). The SeaCat model calculates the attention score, $\sigma_{i}$, for each sentence in a transcript. The sentence preceding and following the agent's first sentence, with the highest score is predicted as the customer's primary question, i.e. the highlighted sentence.
• Figure 2: The SeaCat Model. The model consists of blue blocks, representing tensors, and green blocks representing operators. (a) The neural network is comprised of sentence tensors, $S_{i}$, and a sequence model, $\Sigma$, which outputs $Q^{'}{i}$. The position embedding vector $E{i}$ is combined with $Q^{'}{i}$ to create the sentence embeddings $Q{i}$. Finally, a linear classifier predicts the product/service. (b) The red block in (a) is described in detail. Tensor notation ($S_1;T_n$) refers to the $n$-th token in sentence $S_1$. The sequence model $\Sigma$ processes each word in a sentence using a time-distributed wrapper to handle multiple sentences. (c) The orange block in (a) is explained. A dense layer, $K$, with a softmax activation function is applied to all sentence embedding vectors $Q_{i}$ (via a time-distributed wrapper) to calculate attention scores $\sigma_{i}$. Note that $Q_{i}$ is equivalent to $V_{i}$.
• Figure 3: Illustration of how the Universal Model operates. (a) The top-5 most similar questions are retrieved from the 12K repository for a given customer inquiry (as shown in the first row). (b) The distribution of handling times for the top-100 retrieved similar contacts is displayed. The median of this distribution will be used as the predicted handling time. (c) The distribution of product/service for the top-100 retrieved similar contacts is also shown. The product/service with the highest frequency will be the predicted label. Note: All the sentences and the product/service names are derived from a synthetic research dataset for demonstration purposes, and do not represent any actual customer service data collected from Amazon.