Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations

Weihao Zeng, Dayuan Fu, Keqing He, Yejie Wang, Yukai Xu, Weiran Xu

TL;DR

DivTOD addresses the limited diversity of task-oriented dialogue representations by leveraging a large language model (LLM) as a teacher to generate diverse, domain-consistent responses and distill this diversity into a compact student model. The method consists of three steps: (1) generate diverse system responses using a fill-the-blank prompt, (2) post-filter those responses to align with TOD domain knowledge, and (3) self-train a smaller model to inherit the diversity. Evaluations on intent recognition, dialogue state tracking, dialogue act prediction, and response selection across nine TOD datasets show DivTOD achieving state-of-the-art performance and capturing intrinsic one-to-many diversity, outperforming baselines like FutureTOD and TOD-BERT. This approach demonstrates a scalable way to enhance TOD understanding while maintaining deployment efficiency, with plans to release code and pre-trained models for broader adoption.

Abstract

Language models pre-trained on general text have achieved impressive results in diverse fields. Yet, the distinct linguistic characteristics of task-oriented dialogues (TOD) compared to general text limit the practical utility of existing language models. Current task-oriented dialogue pre-training methods overlook the one-to-many property of conversations, where multiple responses can be appropriate given the same conversation context. In this paper, we propose a novel dialogue pre-training model called DivTOD, which collaborates with LLMs to learn diverse task-oriented dialogue representations. DivTOD guides LLMs in transferring diverse knowledge to smaller models while removing domain knowledge that contradicts task-oriented dialogues. Experiments show that our model outperforms strong TOD baselines on various downstream dialogue tasks and learns the intrinsic diversity of task-oriented dialogues.

DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations

TL;DR

DivTOD addresses the limited diversity of task-oriented dialogue representations by leveraging a large language model (LLM) as a teacher to generate diverse, domain-consistent responses and distill this diversity into a compact student model. The method consists of three steps: (1) generate diverse system responses using a fill-the-blank prompt, (2) post-filter those responses to align with TOD domain knowledge, and (3) self-train a smaller model to inherit the diversity. Evaluations on intent recognition, dialogue state tracking, dialogue act prediction, and response selection across nine TOD datasets show DivTOD achieving state-of-the-art performance and capturing intrinsic one-to-many diversity, outperforming baselines like FutureTOD and TOD-BERT. This approach demonstrates a scalable way to enhance TOD understanding while maintaining deployment efficiency, with plans to release code and pre-trained models for broader adoption.

Abstract

Language models pre-trained on general text have achieved impressive results in diverse fields. Yet, the distinct linguistic characteristics of task-oriented dialogues (TOD) compared to general text limit the practical utility of existing language models. Current task-oriented dialogue pre-training methods overlook the one-to-many property of conversations, where multiple responses can be appropriate given the same conversation context. In this paper, we propose a novel dialogue pre-training model called DivTOD, which collaborates with LLMs to learn diverse task-oriented dialogue representations. DivTOD guides LLMs in transferring diverse knowledge to smaller models while removing domain knowledge that contradicts task-oriented dialogues. Experiments show that our model outperforms strong TOD baselines on various downstream dialogue tasks and learns the intrinsic diversity of task-oriented dialogues.
Paper Structure (25 sections, 6 figures, 10 tables, 1 algorithm)

This paper contains 25 sections, 6 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model
  3. Overall Architecture
  4. Diversifying Task-Oriented Dialogue Representations
  5. Experiment
  6. Pre-training Corpus
  7. Baselines
  8. Implementation Details
  9. Main Results
  10. Qualitative Analysis
  11. Ablation Study of Domain Knowledge Alignment
  12. Advantages of LLMs in Generating Diversified Responses
  13. Quantity of Diverse Dialogues
  14. Few Shot Learning
  15. Zero Shot Learning
  16. ...and 10 more sections

Figures (6)

  • Figure 1: The same context may have multiple appropriate responses in a task-oriented dialogue, which we call one-to-many.
  • Figure 2: Overall architecture of DivTOD.
  • Figure 3: Different Dialogue Cases. Original Dialogues refers to the dialogues from the original TOD dataset. DivTOD's Dialogue refers to the dialogues generated using the complete generating and aligning steps. DivTOD w/o Alignment's Dialogue refers to the dialogues generated after removing domain knowledge alignment.
  • Figure 4: The ablation experiment on the impact of the number of diverse dialogues generated by large language models on TOD.
  • Figure 5: The complete prompt example for generating diversified responses.
  • ...and 1 more figures