Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

Sumanth Balaji, Piyush Mishra, Aashraya Sachdeva, Suraj Agrawal

TL;DR

This work addresses the gap in evaluating LLM-based customer-support agents for strict policy adherence by introducing JourneyBench, a graph-based benchmark that encodes business SOPs as DAGs and measures adherence with the User Journey Coverage Score (UJCS). It presents two agent designs, a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models workflow control, and demonstrates through 703 conversations across three domains that structured orchestration markedly improves policy compliance, enabling smaller models to compete with larger ones. JourneyBench’s pipeline—graph generation, expert validation, BFS-driven user journeys, and scenario data generation—ensures scalable, realistic evaluation while its metrics (Tool Trace Alignment, TCA, and UJCS) quantify pathway fidelity and parameter accuracy. Real-world deployment of DPA in production (6,000+ daily calls) and realism validation (84.37% overall) corroborate the practicality and robustness of policy-aware agents in business settings, signaling a move beyond IVR toward reliable, compliant AI-assisted customer support.

Abstract

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.

Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

TL;DR

This work addresses the gap in evaluating LLM-based customer-support agents for strict policy adherence by introducing JourneyBench, a graph-based benchmark that encodes business SOPs as DAGs and measures adherence with the User Journey Coverage Score (UJCS). It presents two agent designs, a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models workflow control, and demonstrates through 703 conversations across three domains that structured orchestration markedly improves policy compliance, enabling smaller models to compete with larger ones. JourneyBench’s pipeline—graph generation, expert validation, BFS-driven user journeys, and scenario data generation—ensures scalable, realistic evaluation while its metrics (Tool Trace Alignment, TCA, and UJCS) quantify pathway fidelity and parameter accuracy. Real-world deployment of DPA in production (6,000+ daily calls) and realism validation (84.37% overall) corroborate the practicality and robustness of policy-aware agents in business settings, signaling a move beyond IVR toward reliable, compliant AI-assisted customer support.

Abstract

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.
Paper Structure (51 sections, 2 equations, 3 figures, 4 tables)

This paper contains 51 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. JourneyBench Framework
  3. SOP Representation
  4. Node Structure
  5. Synthetic Dataset Curation
  6. Phase 1: Graph Generation and Refinement
  7. Phase 2: Manual Review
  8. Phase 3: User Journey Generation
  9. Phase 4: Scenario Data Generation
  10. Evaluation Metrics
  11. Tool Trace Alignment
  12. Tool Call Accuracy
  13. User Journey Coverage Score (UJCS)
  14. Experimentation
  15. Instantiating Agents with SOP
  16. ...and 36 more sections

Figures (3)

  • Figure 1: Example SOP graph for loan application processing, showing sequential tasks and decision points.
  • Figure 2: Components within a single node: the task description (prompt), available tools for execution, and conditional pathways (edges) that define transitions to the next node based on outcomes.
  • Figure 3: Overview of our data generation pipeline across four phases (P1--P4). Validated SOP Graphs generate numbered user journeys via Breadth-First Search (BFS) traversal, which are then used to create diverse evaluation scenarios for the final benchmark dataset.