Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues

Xiang Luo, Zhiwen Tang, Jin Wang, Xuejie Zhang

TL;DR

DuetSim addresses the challenge of creating diverse yet goal-driven user simulators for task-oriented dialogues by pairing two LLMs: a generator and a verifier. The generator proposes dialogue actions and utterances, while the verifier checks for correctness and provides feedback, enabling iterative refinement. The approach uses zero-shot prompt learning with chain-of-thought reasoning and demonstrates improved goal fulfillment and utterance diversity on MultiWOZ compared with single-LLM baselines and rule-based simulators. Human evaluation confirms higher naturalness and informativeness for the DuetSim variants, and cross-model experiments suggest better generalization when training with DuetSim data. The work delivers a scalable, transferable framework for TODS development and evaluation, with open-source code available.

Abstract

User Simulators play a pivotal role in training and evaluating task-oriented dialogue systems. Traditional user simulators typically rely on human-engineered agendas, resulting in generated responses that often lack diversity and spontaneity. Although large language models (LLMs) exhibit a remarkable capacity for generating coherent and contextually appropriate utterances, they may fall short when tasked with generating responses that effectively guide users towards their goals, particularly in dialogues with intricate constraints and requirements. This paper introduces DuetSim, a novel framework designed to address the intricate demands of task-oriented dialogues by leveraging LLMs. DuetSim stands apart from conventional approaches by employing two LLMs in tandem: one dedicated to response generation and the other focused on verification. This dual LLM approach empowers DuetSim to produce responses that not only exhibit diversity but also demonstrate accuracy and are preferred by human users. We validate the efficacy of our method through extensive experiments conducted on the MultiWOZ dataset, highlighting improvements in response quality and correctness, largely attributed to the incorporation of the second LLM. Our code is accessible at: https://github.com/suntea233/DuetSim.

DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues

TL;DR

DuetSim addresses the challenge of creating diverse yet goal-driven user simulators for task-oriented dialogues by pairing two LLMs: a generator and a verifier. The generator proposes dialogue actions and utterances, while the verifier checks for correctness and provides feedback, enabling iterative refinement. The approach uses zero-shot prompt learning with chain-of-thought reasoning and demonstrates improved goal fulfillment and utterance diversity on MultiWOZ compared with single-LLM baselines and rule-based simulators. Human evaluation confirms higher naturalness and informativeness for the DuetSim variants, and cross-model experiments suggest better generalization when training with DuetSim data. The work delivers a scalable, transferable framework for TODS development and evaluation, with open-source code available.

Abstract

User Simulators play a pivotal role in training and evaluating task-oriented dialogue systems. Traditional user simulators typically rely on human-engineered agendas, resulting in generated responses that often lack diversity and spontaneity. Although large language models (LLMs) exhibit a remarkable capacity for generating coherent and contextually appropriate utterances, they may fall short when tasked with generating responses that effectively guide users towards their goals, particularly in dialogues with intricate constraints and requirements. This paper introduces DuetSim, a novel framework designed to address the intricate demands of task-oriented dialogues by leveraging LLMs. DuetSim stands apart from conventional approaches by employing two LLMs in tandem: one dedicated to response generation and the other focused on verification. This dual LLM approach empowers DuetSim to produce responses that not only exhibit diversity but also demonstrate accuracy and are preferred by human users. We validate the efficacy of our method through extensive experiments conducted on the MultiWOZ dataset, highlighting improvements in response quality and correctness, largely attributed to the incorporation of the second LLM. Our code is accessible at: https://github.com/suntea233/DuetSim.
Paper Structure (19 sections, 3 equations, 3 figures, 7 tables)

This paper contains 19 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. User Simulator
  4. Large Language Models and Prompt Learning
  5. DuetSim
  6. Prompt Learning for DuetSim
  7. Generator-Verifier Interaction
  8. Chain-of-thought Response Generation
  9. Experiments
  10. Dataset
  11. Evaluation Metrics
  12. Implementation Details
  13. Baselines
  14. Main Results
  15. Ablation Study
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Different Architectures for LLM-based User Simulators
  • Figure 2: Details of DuetSim.
  • Figure 3: Dialogues generated by the model w/ and w/o verifier.