Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DARD: A Multi-Agent Approach for Task-Oriented Dialog Systems

Aman Gupta, Anirudh Ravichandran, Ziji Zhang, Swair Shah, Anurag Beniwal, Narayanan Sadagopan

TL;DR

DARD (Domain Assigned Response Delegation), a multi-agent conversational system capable of successfully handling multi-domain dialogs, is proposed and evaluated using the well-established MultiWOZ benchmark, achieving state-of-the-art performance.

Abstract

Task-oriented dialogue systems are essential for applications ranging from customer service to personal assistants and are widely used across various industries. However, developing effective multi-domain systems remains a significant challenge due to the complexity of handling diverse user intents, entity types, and domain-specific knowledge across several domains. In this work, we propose DARD (Domain Assigned Response Delegation), a multi-agent conversational system capable of successfully handling multi-domain dialogs. DARD leverages domain-specific agents, orchestrated by a central dialog manager agent. Our extensive experiments compare and utilize various agent modeling approaches, combining the strengths of smaller fine-tuned models (Flan-T5-large & Mistral-7B) with their larger counterparts, Large Language Models (LLMs) (Claude Sonnet 3.0). We provide insights into the strengths and limitations of each approach, highlighting the benefits of our multi-agent framework in terms of flexibility and composability. We evaluate DARD using the well-established MultiWOZ benchmark, achieving state-of-the-art performance by improving the dialogue inform rate by 6.6% and the success rate by 4.1% over the best-performing existing approaches. Additionally, we discuss various annotator discrepancies and issues within the MultiWOZ dataset and its evaluation system.

DARD: A Multi-Agent Approach for Task-Oriented Dialog Systems

TL;DR

DARD (Domain Assigned Response Delegation), a multi-agent conversational system capable of successfully handling multi-domain dialogs, is proposed and evaluated using the well-established MultiWOZ benchmark, achieving state-of-the-art performance.

Abstract

Task-oriented dialogue systems are essential for applications ranging from customer service to personal assistants and are widely used across various industries. However, developing effective multi-domain systems remains a significant challenge due to the complexity of handling diverse user intents, entity types, and domain-specific knowledge across several domains. In this work, we propose DARD (Domain Assigned Response Delegation), a multi-agent conversational system capable of successfully handling multi-domain dialogs. DARD leverages domain-specific agents, orchestrated by a central dialog manager agent. Our extensive experiments compare and utilize various agent modeling approaches, combining the strengths of smaller fine-tuned models (Flan-T5-large & Mistral-7B) with their larger counterparts, Large Language Models (LLMs) (Claude Sonnet 3.0). We provide insights into the strengths and limitations of each approach, highlighting the benefits of our multi-agent framework in terms of flexibility and composability. We evaluate DARD using the well-established MultiWOZ benchmark, achieving state-of-the-art performance by improving the dialogue inform rate by 6.6% and the success rate by 4.1% over the best-performing existing approaches. Additionally, we discuss various annotator discrepancies and issues within the MultiWOZ dataset and its evaluation system.

Paper Structure

This paper contains 20 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. MultiWOZ 2.2 Dataset
  4. Scoring Metrics
  5. Approach
  6. DST Prediction
  7. Response Generation Pipeline
  8. Results
  9. Discussions
  10. Analysis of Claude's DST performance
  11. Single Agent Vs Multi-Agent Approaches
  12. Fine-tuned Vs Prompted Models
  13. Conclusion
  14. Future Work & Limitations
  15. Appendix / supplemental material
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of DARD for end-to-end response generation on MultiWOZ. The current diagram shows a conversation assigned to the restaurant agent but in general, it can be assigned to any of the domain agents
  • Figure 2: Two samples from the MultiWOZ 2.2 dataset which show how different annotators respond to the users