Anchorage: Visual Analysis of Satisfaction in Customer Service Videos via Anchor Events

TL;DR

This work addresses automatic evaluation of customer satisfaction from video-based service interactions, where self-reported metrics are scarce and events are sparsely distributed. It introduces Anchorage, a visual analytics system that uses two types of anchors—operational anchors (events in service procedures) and behavioral anchors (multimodal cues)—coupled with PCA and Markov-based anomaly detection to structure videos and highlight turning points. The system provides a buoy-style visualization and multiple coordinated views to summarize satisfaction, navigate to salient segments, and verify automatic scores with machine logs. Case studies and a user study demonstrate that incorporating event contexts improves satisfaction evaluation, enhances discrimination among satisfaction types, and increases user trust while reducing analysis time, with potential applicability to unlabelled, unstructured videos paired with sequential records.

Abstract

Delivering customer services through video communications has brought new opportunities to analyze customer satisfaction for quality management. However, due to the lack of reliable self-reported responses, service providers are troubled by the inadequate estimation of customer services and the tedious investigation into multimodal video recordings. We introduce Anchorage, a visual analytics system to evaluate customer satisfaction by summarizing multimodal behavioral features in customer service videos and revealing abnormal operations in the service process. We leverage the semantically meaningful operations to introduce structured event understanding into videos which help service providers quickly navigate to events of their interest. Anchorage supports a comprehensive evaluation of customer satisfaction from the service and operation levels and efficient analysis of customer behavioral dynamics via multifaceted visualization views. We extensively evaluate Anchorage through a case study and a carefully-designed user study. The results demonstrate its effectiveness and usability in assessing customer satisfaction using customer service videos. We found that introducing event contexts in assessing customer satisfaction can enhance its performance without compromising annotation precision. Our approach can be adapted in situations where unlabelled and unstructured videos are collected along with sequential records.

Figures (9)

• Figure 1: A customer service example modified from a customer service dialogue dataset in online shopping intro_challenge_chen_2021. (A) The agent guidelines outline the typical procedure. (B) The service procedure is interwoven with the interactions and operations from the agent and the client. (C) The service record logs the executed operations with metadata such as timestamps.
• Figure 2: The system architecture of Anchorage contains three modules: the data storage, the anchor generation module, and the visual interface.
• Figure 3: The interface of Anchorage magnifies customer satisfaction metrics with greater sequential and temporal resolutions. The service overview (A) shows the primitive satisfaction scores based on clients' responses and the smoothness of service procedures. The anchor exploration view (B) displays the customer satisfaction progression in a service segmented by operations and highlights anomalies. The multimodal feature navigation view (C) provides detailed multimodal information for verifying the satisfaction scores and supports interactive navigation to the service video view (D).
• Figure 4: Anchorage employs the buoy metaphor to summarize multimodal satisfaction evaluation with procedural understanding. (A) shows the four quadrants heuristic of the buoy chart. (B) is the lateral buoy chart that progressively plots the multimodal satisfaction scores for the five operations ($e_{1-5}$). (C) is the buoy chart that summarizes all operations of a service in a single graph. (D-E) are the alternative designs of (C).
• Figure 5: The unsatisfied case shows a client feeling annoyed by the repeated operations and the inattentive agent.