Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation

Rishikesh Devanathan, Varun Nathan, Ayush Kumar

TL;DR

This work benchmarks multiple generation strategies guided by structured supervision on call attributes across multiple languages and shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism.

Abstract

Synthetic data is increasingly critical for contact centers, where privacy constraints and data scarcity limit the availability of real conversations. However, generating synthetic dialogues that are realistic and useful for downstream applications remains challenging. In this work, we benchmark multiple generation strategies guided by structured supervision on call attributes (Intent Summaries, Topic Flows, and Quality Assurance (QA) Forms) across multiple languages. To test downstream utility, we evaluate synthetic transcripts on an automated quality assurance (AutoQA) task, finding that prompts optimized on real transcripts consistently outperform those optimized on synthetic transcripts. These results suggest that current synthetic transcripts fall short in capturing the full realism of real agent-customer interactions. To highlight these downstream gaps, we introduce a diagnostic evaluation framework comprising 17 metrics across four dimensions: (1) Emotional and Sentiment Arcs, (2) Linguistic Complexity, (3) Interaction Style, and (4) Conversational Properties. Our analysis shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism. Together, these results highlight the importance of diagnostic, metric-driven evaluation for synthetic conversation generation intended for downstream applications.

Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation

TL;DR

This work benchmarks multiple generation strategies guided by structured supervision on call attributes across multiple languages and shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism.

Abstract

Synthetic data is increasingly critical for contact centers, where privacy constraints and data scarcity limit the availability of real conversations. However, generating synthetic dialogues that are realistic and useful for downstream applications remains challenging. In this work, we benchmark multiple generation strategies guided by structured supervision on call attributes (Intent Summaries, Topic Flows, and Quality Assurance (QA) Forms) across multiple languages. To test downstream utility, we evaluate synthetic transcripts on an automated quality assurance (AutoQA) task, finding that prompts optimized on real transcripts consistently outperform those optimized on synthetic transcripts. These results suggest that current synthetic transcripts fall short in capturing the full realism of real agent-customer interactions. To highlight these downstream gaps, we introduce a diagnostic evaluation framework comprising 17 metrics across four dimensions: (1) Emotional and Sentiment Arcs, (2) Linguistic Complexity, (3) Interaction Style, and (4) Conversational Properties. Our analysis shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism. Together, these results highlight the importance of diagnostic, metric-driven evaluation for synthetic conversation generation intended for downstream applications.

Paper Structure

This paper contains 52 sections, 7 equations, 2 figures, 45 tables.

Table of Contents

  1. Introduction and Related Work
  2. Methodology
  3. Generation Methodology
  4. Direct Generation
  5. Chunked Enhancement
  6. Characteristic-Aware Enhancement
  7. Evaluation Framework
  8. Evaluation on Downstream Task
  9. Experimental Setup
  10. Dataset
  11. Models
  12. Baselines
  13. Prompt Optimization For Generation
  14. Implementation Overview
  15. Results
  16. ...and 37 more sections

Figures (2)

  • Figure 1: Diagram depicting the three generation strategies. Section in red depicts Direct generation where transcripts are generated by simply conditioning on the input call attributes $(A)$. Sections in purple and orange depict the Chunked and Characteristic Aware enhancement stages respectively.
  • Figure 2: Comparison of AutoQA performance using real and synthetic transcripts for prompt optimization.