Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Schema Augmentation for Zero-Shot Domain Adaptation in Dialogue State Tracking

Christopher Richardson, Roshan Sharma, Neeraj Gaur, Parisa Haghani, Anirudh Sundar, Bhuvana Ramabhadran

TL;DR

This paper tackles zero-shot domain adaptation in dialogue state tracking by shifting focus from prompt engineering to data augmentation of the schema. It introduces Schema Augmentation with two variants (SSA and ESA) to create diverse training prompts and evaluates end-to-end DST fine-tuning on instruction-tuned models, demonstrating substantial gains in Target Goal Accuracy on unseen domains while maintaining strong overall DST performance. A new Target Goal Accuracy metric is proposed to better reflect true cross-domain transfer, mitigating issues with empty target substates observed in prior evaluation setups. The findings show that encoding-based ESA typically yields the strongest improvements across both MultiWOZ and SpokenWOZ, indicating greater robustness to schema variations and improved generalization to unseen domains in task-oriented dialogue systems.

Abstract

Zero-shot domain adaptation for dialogue state tracking (DST) remains a challenging problem in task-oriented dialogue (TOD) systems, where models must generalize to target domains unseen at training time. Current large language model approaches for zero-shot domain adaptation rely on prompting to introduce knowledge pertaining to the target domains. However, their efficacy strongly depends on prompt engineering, as well as the zero-shot ability of the underlying language model. In this work, we devise a novel data augmentation approach, Schema Augmentation, that improves the zero-shot domain adaptation of language models through fine-tuning. Schema Augmentation is a simple but effective technique that enhances generalization by introducing variations of slot names within the schema provided in the prompt. Experiments on MultiWOZ and SpokenWOZ showed that the proposed approach resulted in a substantial improvement over the baseline, in some experiments achieving over a twofold accuracy gain over unseen domains while maintaining equal or superior performance over all domains.

Schema Augmentation for Zero-Shot Domain Adaptation in Dialogue State Tracking

TL;DR

This paper tackles zero-shot domain adaptation in dialogue state tracking by shifting focus from prompt engineering to data augmentation of the schema. It introduces Schema Augmentation with two variants (SSA and ESA) to create diverse training prompts and evaluates end-to-end DST fine-tuning on instruction-tuned models, demonstrating substantial gains in Target Goal Accuracy on unseen domains while maintaining strong overall DST performance. A new Target Goal Accuracy metric is proposed to better reflect true cross-domain transfer, mitigating issues with empty target substates observed in prior evaluation setups. The findings show that encoding-based ESA typically yields the strongest improvements across both MultiWOZ and SpokenWOZ, indicating greater robustness to schema variations and improved generalization to unseen domains in task-oriented dialogue systems.

Abstract

Zero-shot domain adaptation for dialogue state tracking (DST) remains a challenging problem in task-oriented dialogue (TOD) systems, where models must generalize to target domains unseen at training time. Current large language model approaches for zero-shot domain adaptation rely on prompting to introduce knowledge pertaining to the target domains. However, their efficacy strongly depends on prompt engineering, as well as the zero-shot ability of the underlying language model. In this work, we devise a novel data augmentation approach, Schema Augmentation, that improves the zero-shot domain adaptation of language models through fine-tuning. Schema Augmentation is a simple but effective technique that enhances generalization by introducing variations of slot names within the schema provided in the prompt. Experiments on MultiWOZ and SpokenWOZ showed that the proposed approach resulted in a substantial improvement over the baseline, in some experiments achieving over a twofold accuracy gain over unseen domains while maintaining equal or superior performance over all domains.

Paper Structure

This paper contains 25 sections, 8 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Methods
  5. Target Goal Accuracy
  6. Joint Goal Accuracy (JGA):
  7. Target Goal Accuracy (TGA):
  8. Schema Augmentation
  9. Experiments
  10. Modeling
  11. Datasets and Baselines
  12. Main Results
  13. SpokenWOZ Results
  14. Conclusion
  15. Appendix
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Examples of domain and slot replacements for both Schema Augmentation types: SSA (blue box) and ESA (green box).
  • Figure 2: Ablation results for MultiWOZ 2.2. SSA and ESA are our Schema Augmentation methods. No Aug means fine-tuning gemma-2-9b-it without data augmentation, to illustrate the effects of Schema Augmentation.