Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Faceted Evaluation of Tool-Augmented Dialogue Systems

Zhaoyi Joey Hou, Tanya Shourya, Yingfan Wang, Shamik Roy, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah

TL;DR

This work tackles the challenge of evaluating tool-augmented dialogue systems by showing that user-satisfaction alone misses critical errors arising from interactions among users, agents, and tools. It introduces TRACE, a benchmark of 516 synthetic, multi-turn conversations across 26 error cases, and SCOPE, a four-stage evaluation framework that discovers evaluation areas, generates weighted rubrics, and estimates conversation quality using a normalized, weighted scoring scheme. Empirical results demonstrate that SCOPE significantly outperforms the prior SPUR baseline, with substantial gains on hard negative cases where user satisfaction is misleading, highlighting the framework's ability to detect subtle tool-use and agent-interaction failures. The work lays a foundation for more robust and trustworthy tool-augmented dialogue systems and discusses limitations related to synthetic data and labeling quality, offering directions for future real-world, multilingual evaluation and release of resources.

Abstract

Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation methods assess either user satisfaction or agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a benchmark of systematically synthesized tool-augmented conversations covering diverse error cases, and SCOPE, an evaluation framework that automatically discovers diverse error patterns and evaluation rubrics in tool-augmented dialogues. Experiments show SCOPE significantly outperforms the baseline, particularly on challenging cases where user satisfaction signals are misleading.

Multi-Faceted Evaluation of Tool-Augmented Dialogue Systems

TL;DR

This work tackles the challenge of evaluating tool-augmented dialogue systems by showing that user-satisfaction alone misses critical errors arising from interactions among users, agents, and tools. It introduces TRACE, a benchmark of 516 synthetic, multi-turn conversations across 26 error cases, and SCOPE, a four-stage evaluation framework that discovers evaluation areas, generates weighted rubrics, and estimates conversation quality using a normalized, weighted scoring scheme. Empirical results demonstrate that SCOPE significantly outperforms the prior SPUR baseline, with substantial gains on hard negative cases where user satisfaction is misleading, highlighting the framework's ability to detect subtle tool-use and agent-interaction failures. The work lays a foundation for more robust and trustworthy tool-augmented dialogue systems and discusses limitations related to synthetic data and labeling quality, offering directions for future real-world, multilingual evaluation and release of resources.

Abstract

Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation methods assess either user satisfaction or agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a benchmark of systematically synthesized tool-augmented conversations covering diverse error cases, and SCOPE, an evaluation framework that automatically discovers diverse error patterns and evaluation rubrics in tool-augmented dialogues. Experiments show SCOPE significantly outperforms the baseline, particularly on challenging cases where user satisfaction signals are misleading.
Paper Structure (38 sections, 23 figures, 15 tables)

This paper contains 38 sections, 23 figures, 15 tables.

Table of Contents

  1. Introduction
  2. TRACE Benchmark
  3. Characterizing Tool-Use Conversations
  4. Conversation Generation
  5. Filtering Synthetic Conversations
  6. Evaluation Framework
  7. Baseline: SPUR Lin2024-dr
  8. Proposed Framework: SCOPE
  9. Experimental Evaluation
  10. Dataset and Experimental Setup
  11. Results and Ablations
  12. Conclusion
  13. Related Work
  14. Synthetic Data Generation
  15. Conversation Generation Implementation
  16. ...and 23 more sections

Figures (23)

  • Figure 1: The proposed SCOPE framework.
  • Figure 2: Conversation synthesis: conversation generation prompt
  • Figure 3: Conversation synthesis: random name generator prompt
  • Figure 4: Example of generated conversation. WrongTool / Silent: The agent grabs the completely wrong tool and takes incorrect actions, yet still pretends everything went perfectly, so the user walks away pleased—but in reality, the requested task never happened the way they thought. User Satisfaction: True; Tool Execution Correctness: incorrect due to agent; Agent Performance: not appropriate.
  • Figure 5: Example of generated conversation. Context Amnesia – After successfully invoking a tool, the agent discards the earlier user context and redundantly requests information, causing irritation and wasted turns. User Satisfaction: False; Tool Execution Correctness: correct; Agent Performance: not appropriate.
  • ...and 18 more figures