Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UNO-DST: Leveraging Unlabelled Data in Zero-Shot Dialogue State Tracking

Chuang Li, Yan Zhang, Min-Yen Kan, Haizhou Li

TL;DR

UNO-DST tackles zero-shot dialogue state tracking by leveraging unlabeled target-domain data through a dual-task framework: a main task that predicts slot values and an auxiliary task that generates slot types. These tasks are trained jointly on source domains and then augmented with cycle-consistent self-training on the target domain to select high-quality samples, effectively turning zero-shot DST into a few-shot setting. The approach yields consistent improvements on MultiWOZ and SGD datasets, with additional gains realized when extending to large language models such as ChatGPT via conversational and in-context learning. The method is model-agnostic and highlights the potential for automatic label creation, unseen/new slot-type generation, and improved data efficiency in DST systems.

Abstract

Previous zero-shot dialogue state tracking (DST) methods only apply transfer learning, ignoring unlabelled data in the target domain. We transform zero-shot DST into few-shot DST by utilising such unlabelled data via joint and self-training methods. Our method incorporates auxiliary tasks that generate slot types as inverse prompts for main tasks, creating slot values during joint training. Cycle consistency between these two tasks enables the generation and selection of quality samples in unknown target domains for subsequent fine-tuning. This approach also facilitates automatic label creation, thereby optimizing the training and fine-tuning of DST models. We demonstrate this method's effectiveness on general language models in zero-shot scenarios, improving average joint goal accuracy by 8% across all domains in MultiWOZ.

UNO-DST: Leveraging Unlabelled Data in Zero-Shot Dialogue State Tracking

TL;DR

UNO-DST tackles zero-shot dialogue state tracking by leveraging unlabeled target-domain data through a dual-task framework: a main task that predicts slot values and an auxiliary task that generates slot types. These tasks are trained jointly on source domains and then augmented with cycle-consistent self-training on the target domain to select high-quality samples, effectively turning zero-shot DST into a few-shot setting. The approach yields consistent improvements on MultiWOZ and SGD datasets, with additional gains realized when extending to large language models such as ChatGPT via conversational and in-context learning. The method is model-agnostic and highlights the potential for automatic label creation, unseen/new slot-type generation, and improved data efficiency in DST systems.

Abstract

Previous zero-shot dialogue state tracking (DST) methods only apply transfer learning, ignoring unlabelled data in the target domain. We transform zero-shot DST into few-shot DST by utilising such unlabelled data via joint and self-training methods. Our method incorporates auxiliary tasks that generate slot types as inverse prompts for main tasks, creating slot values during joint training. Cycle consistency between these two tasks enables the generation and selection of quality samples in unknown target domains for subsequent fine-tuning. This approach also facilitates automatic label creation, thereby optimizing the training and fine-tuning of DST models. We demonstrate this method's effectiveness on general language models in zero-shot scenarios, improving average joint goal accuracy by 8% across all domains in MultiWOZ.
Paper Structure (21 sections, 5 equations, 7 figures, 6 tables)

This paper contains 21 sections, 5 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Task Definition
  5. Auxiliary task
  6. Joint training with auxiliary tasks
  7. Self-training with auxiliary tasks
  8. Oracular selection for zero-shot DST
  9. Experiments and Datasets
  10. Results
  11. Joint training results
  12. Self-training results
  13. Discussion
  14. Oracular selection
  15. Unseen slot type prediction
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Examples of zero-shot methods in DST.
  • Figure 2: Cycle consistency in DST.
  • Figure 3: Overview of UNO-DST which consists of two periods: 1) joint training for both task A (slot value prediction) and B (slot type prediction), 2) self-training in the target domain. Step 1: Generation of slot values and types; Step 2: Selection of good samples with cycle consistency; Step 3: Fine-turning the LM with selected samples.
  • Figure 4: Gains by joint and self-training stages of UNO-DST on the "t5-QA" checkpoint. We show the results of oracular selection (Upper-bound) in each domain for relative comparison.
  • Figure 5: Slot Accuracy for (grey) seen and (red) unseen slot types in the hotel domain.
  • ...and 2 more figures