ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling

TL;DR

ESAinsTOD, a unified End-to-end Schema-Aware Instruction-tuning framework for general Task-Oriented Dialog modeling, introduces a structured methodology to go beyond simply fine-tuning Large Language Models, enabling flexible adaptation to various dialogue task flows and schemas.

Abstract

Existing end-to-end modeling methods for modular task-oriented dialog systems are typically tailored to specific datasets, making it challenging to adapt to new dialog scenarios. In this work, we propose ESAinsTOD, a unified End-to-end Schema-Aware Instruction-tuning framework for general Task-Oriented Dialog modeling. This framework introduces a structured methodology to go beyond simply fine-tuning Large Language Models (LLMs), enabling flexible adaptation to various dialogue task flows and schemas. Specifically, we leverage full-parameter fine-tuning of LLMs and introduce two alignment mechanisms to make the resulting system both instruction-aware and schema-aware: (i) instruction alignment, which ensures that the system faithfully follows task instructions to complete various task flows from heterogeneous TOD datasets; and (ii) schema alignment, which encourages the system to make predictions adhering to the specified schema. In addition, we employ session-level end-to-end modeling, which allows the system to access the results of previously executed task flows within the dialogue history, to bridge the gap between the instruction-tuning paradigm and the real-world application of TOD systems. Empirical results show that while a fine-tuned LLM serves as a strong baseline, our structured approach provides significant additional benefits. In particular, our findings indicate that: (i) ESAinsTOD outperforms state-of-the-art models by a significant margin on end-to-end task-oriented dialog modeling benchmarks: CamRest676, In-Car and MultiWOZ; (ii) more importantly, it exhibits superior generalization capabilities across various low-resource settings, with the proposed alignment mechanisms significantly enhancing zero-shot performance; and (iii) our instruction-tuning paradigm substantially improves the model's robustness against data noise and cascading errors.

Figures (5)

• Figure 1: Comparison between conventional PCMs and our instruction-tuned LLM for task-oriented dialog systems. In Figure (a), PCM A and PCM B are derived by fine-tuning pre-trained models on specific corpora A and B, respectively. Each PCM is tailored for a particular workflow and lacks the flexibility to apply to other dialogue scenarios. In contrast, as depicted in Figure (b), the TOD Instruction-tuned LLM is developed by fine-tuning a large language model on multiple corpora, employing both instruction-aware and schema-aware mechanisms. This design enables it to generalize effectively to rare or unseen dialogue scenarios.
• Figure 2: Illustration of the instruction-tuning paradigm for task-oriented dialog systems
• Figure 3: Low-resource end-to-end evaluation on MultiWOZ 2.0
• Figure 4: Evaluation results of ESAinsTOD trained on varying percentages of our instruction-tuning corpus with or without schema information
• Figure 5: Results of ESAinsTOD on MultiWOZ 2.1 with varying context window sizes