Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback

Yisha Wu, Cen Mia Zhao, Yuanpei Cao, Xiaoqing Su, Yashar Mehdad, Mindy Ji, Claire Na Cheng

TL;DR

This work addresses the cost of maintaining context in multi-channel customer-support conversations by proposing an incremental summarization system that generates concise notes online via progressive note-taking and a fine-tuned Mixtral-8x7B model. A DeBERTa-based bullet relevance classifier filters non-essential updates, while an Agent Edits Learning Framework integrates real-time agent feedback online and offline retraining to continually improve the system. In production, the approach yields a 3% reduction in average case handling time, with up to 9% reductions on complex cases, and maintains high agent satisfaction across English, French, and Spanish deployments. Evaluation combines offline LLM-judge assessments and a Diff-in-Diff online study, demonstrating improved summary quality and substantial operational efficiency at scale, with insights into multilingual performance and deployment considerations.

Abstract

We introduce an incremental summarization system for customer support agents that intelligently determines when to generate concise bullet notes during conversations, reducing agents' context-switching effort and redundant review. Our approach combines a fine-tuned Mixtral-8x7B model for continuous note generation with a DeBERTa-based classifier to filter trivial content. Agent edits refine the online notes generation and regularly inform offline model retraining, closing the agent edits feedback loop. Deployed in production, our system achieved a 3% reduction in case handling time compared to bulk summarization (with reductions of up to 9% in highly complex cases), alongside high agent satisfaction ratings from surveys. These results demonstrate that incremental summarization with continuous feedback effectively enhances summary quality and agent productivity at scale.

Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback

TL;DR

This work addresses the cost of maintaining context in multi-channel customer-support conversations by proposing an incremental summarization system that generates concise notes online via progressive note-taking and a fine-tuned Mixtral-8x7B model. A DeBERTa-based bullet relevance classifier filters non-essential updates, while an Agent Edits Learning Framework integrates real-time agent feedback online and offline retraining to continually improve the system. In production, the approach yields a 3% reduction in average case handling time, with up to 9% reductions on complex cases, and maintains high agent satisfaction across English, French, and Spanish deployments. Evaluation combines offline LLM-judge assessments and a Diff-in-Diff online study, demonstrating improved summary quality and substantial operational efficiency at scale, with insights into multilingual performance and deployment considerations.

Abstract

We introduce an incremental summarization system for customer support agents that intelligently determines when to generate concise bullet notes during conversations, reducing agents' context-switching effort and redundant review. Our approach combines a fine-tuned Mixtral-8x7B model for continuous note generation with a DeBERTa-based classifier to filter trivial content. Agent edits refine the online notes generation and regularly inform offline model retraining, closing the agent edits feedback loop. Deployed in production, our system achieved a 3% reduction in case handling time compared to bulk summarization (with reductions of up to 9% in highly complex cases), alongside high agent satisfaction ratings from surveys. These results demonstrate that incremental summarization with continuous feedback effectively enhances summary quality and agent productivity at scale.

Paper Structure

This paper contains 54 sections, 1 equation, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Long multi‑source inputs
  3. High accuracy demands.
  4. Our Contributions
  5. Related Works
  6. Managing long multi-source inputs
  7. Ensuring High Accuracy
  8. Evaluation with AI
  9. System Architecture and Methods
  10. Progressive Note-Taking with Quality Control
  11. LLM-based Summarization with Continuous Bullet Generation
  12. Bullet Relevance Classifier for Filtering
  13. Agent Edits Learning Framework
  14. Online Interaction Updates
  15. Offline Model Refinement
  16. ...and 39 more sections

Figures (6)

  • Figure 1: System Architecture
  • Figure 2: Progressive Note-Taking Workflow.
  • Figure 3: Dual-path Agent Edit Feedback Workflow.
  • Figure 4: Confusion Matrix for the bullet Classifier
  • Figure 5: ROC AUC for the bullet Classifier
  • ...and 1 more figures