CXMArena: Unified Dataset to benchmark performance in realistic CXM Scenarios

TL;DR

CXMArena addresses the evaluation gap for AI in Customer Experience Management by offering a large-scale synthetic benchmark that tightly couples brand-specific knowledge bases with grounded, noisy conversations. The authors introduce a scalable LLM-driven data generation pipeline to create Information and Issue KBs, simulated customer-agent dialogues, and five operational tasks: Knowledge Base Refinement, Intent Prediction, Agent Quality Adherence, Article Search, and Multi-turn RAG with Integrated Tools. Baseline experiments across these tasks reveal significant challenges for current models in KB maintenance, domain-grounded retrieval, and tool-enabled dialogue, underscoring the need for integrated, end-to-end CXM pipelines. By providing a privacy-safe, domain-specific benchmark and accompanying validation, CXMArena aims to catalyze development of robust AI solutions with practical utility in real contact-center environments, along with plans to extend to more domains and languages.

Abstract

Large Language Models (LLMs) hold immense potential for revolutionizing Customer Experience Management (CXM), particularly in contact center operations. However, evaluating their practical utility in complex operational environments is hindered by data scarcity (due to privacy concerns) and the limitations of current benchmarks. Existing benchmarks often lack realism, failing to incorporate deep knowledge base (KB) integration, real-world noise, or critical operational tasks beyond conversational fluency. To bridge this gap, we introduce CXMArena, a novel, large-scale synthetic benchmark dataset specifically designed for evaluating AI in operational CXM contexts. Given the diversity in possible contact center features, we have developed a scalable LLM-powered pipeline that simulates the brand's CXM entities that form the foundation of our datasets-such as knowledge articles including product specifications, issue taxonomies, and contact center conversations. The entities closely represent real-world distribution because of controlled noise injection (informed by domain experts) and rigorous automated validation. Building on this, we release CXMArena, which provides dedicated benchmarks targeting five important operational tasks: Knowledge Base Refinement, Intent Prediction, Agent Quality Adherence, Article Search, and Multi-turn RAG with Integrated Tools. Our baseline experiments underscore the benchmark's difficulty: even state of the art embedding and generation models achieve only 68% accuracy on article search, while standard embedding methods yield a low F1 score of 0.3 for knowledge base refinement, highlighting significant challenges for current models necessitating complex pipelines and solutions over conventional techniques.

Figures (13)

• Figure 1: Workflow diagram for the CXMArena creation process, showing the steps from initial KB generation to the final extraction of benchmark task data. This is a high-level overview of our data generation pipeline. The nuanced details of KB index and content generation which ensure real-world distribution is covered in Appendix A.
• Figure 2: The first diagram shows the accuracy scores of multiple models averaged across analytics tasks, i.e, Intent Prediction and Agent Quality Adherence while the second diagrams benchmarks different embedding techniques for tasks like Article Search, Multi Turn RAG and Knowledge Base Refinement.
• Figure 3: Example of conversation-level metadata associated with a simulated dialogue in CXMArena, detailing the Knowledge Bases (KBs), agent persona parameters (e.g., quality metrics, tools/skills), and customer context used during generation.
• Figure 4: Illustration of message-level metadata within CXMArena, showing how individual agent messages are grounded to specific Knowledge Base (KB) passages or linked to tool usage events.
• Figure 5: Example of a simulated customer-agent conversation generated for the CXMArena dataset.