ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue System

TL;DR

This work addresses the gap in evaluating agentic, memory-enabled task-oriented dialogue by introducing ATOD, a synthetic benchmark that encapsulates multi-goal concurrency, interleaving, long-horizon memory, asynchronous execution, and proactivity. Built on ATOD, ATOD-Eval provides a holistic, reproducible set of metrics for offline and online evaluation, including a novel agentic memory-based evaluator that tracks evolving goal trajectories through a dual memory store. Experimental results show the memory-based evaluator yields superior goal-detection and status-tracking accuracy with improved efficiency compared to LLM-based and other baselines, especially in complex, long-horizon dialogues. The framework enables robust assessment of advanced TOD capabilities and offers a scalable foundation for evaluating next-generation, memory-augmented dialogue systems in real-world scenarios.

Abstract

Recent advances in task-oriented dialogue (TOD) systems, driven by large language models (LLMs) with extensive API and tool integration, have enabled conversational agents to coordinate interleaved goals, maintain long-horizon context, and act proactively through asynchronous execution. These capabilities extend beyond traditional TOD systems, yet existing benchmarks lack systematic support for evaluating such agentic behaviors. To address this gap, we introduce ATOD, a benchmark and synthetic dialogue generation pipeline that produces richly annotated conversations requiring long-term reasoning. ATOD captures key characteristics of advanced TOD, including multi-goal coordination, dependency management, memory, adaptability, and proactivity. Building on ATOD, we propose ATOD-Eval, a holistic evaluation framework that translates these dimensions into fine-grained metrics and supports reproducible offline and online evaluation. We further present a strong agentic memory-based evaluator for benchmarking on ATOD. Experiments show that ATOD-Eval enables comprehensive assessment across task completion, agentic capability, and response quality, and that the proposed evaluator offers a better accuracy-efficiency tradeoff compared to existing memory- and LLM-based approaches under this evaluation setting.

Figures (4)

• Figure 1: ATOD dataset curation pipeline. (a) Co-occurrence Graph & Trajectory Sampling (§\ref{['sec:graph_construction']}): Construct a goal co-occurrence graph from seed dialogues and sample diverse multi-goal trajectories via random walks; (b) Trajectory Annotation & Dialogue Generation (§\ref{['sec:annotation_classification']}–§\ref{['sec:dialog_generation']}): An LLM annotates slot values, inter-goal dependencies, and complexity, then generates agentic multi-turn dialogues conditioned on the trajectories; (c) Goal Status Annotation (§\ref{['sec:status_annotation']}): At each turn, an LLM labels active goals and updates lifecycle states, enabling fine-grained tracking of dialogue progress.
• Figure 2: Architecture of the agentic memory system. (a) Turn-level pipeline for goal extraction, existence checking, updating/inserting, and proactive auditing. (b) Dual memory store with symbolic metadata and semantic embeddings. (c) Dependency graph evolution when inserting new goals, with explicit links and status transitions.
• Figure 3: Goal detection F1 (top) and status tracking accuracy (bottom) vs. normalized dialogue progress (0–100%) under Medium and Complex settings.
• Figure 4: Per-turn update latency (top) and token usage (bottom) across methods. Latency is reported as mean per-turn time with range bars; token usage reports mean input and output tokens per turn under Medium and Complex settings.