CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems

TL;DR

The paper tackles the high labeling cost in task-oriented dialog (ToD) by introducing Comprehensive Instruction (CIns), which augments short prompts with task-specific Definition and Constraint to better leverage pre-trained language models. Implemented on a unified T5 Seq2Seq framework, CIns is instantiated for intent classification, dialog state tracking, and natural language generation, and evaluated under realistic few-shot conditions with small validation sets. Across three ToD tasks, CIns consistently outperforms standard prompting and vanilla fine-tuning, with substantial gains in very low-resource settings and competitive performance near full supervision in generation tasks. This work demonstrates that explicit, task-aware instructions can dramatically improve sample-efficient ToD systems and guides future research on instruction-based strategies for PLMs.

Abstract

As labeling cost for different modules in task-oriented dialog (ToD) systems is high, a major challenge in practice is to learn different tasks with the least amount of labeled data. Recently, prompting methods over pre-trained language models (PLMs) have shown promising results for few-shot learning in ToD. To better utilize the power of PLMs, this paper proposes Comprehensive Instruction (CINS) that exploits PLMs with extra task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in ToD, i.e. intent classification, dialog state tracking, and natural language generation. A sequence-to-sequence model (T5) is adopted to solve these three tasks in a unified framework. Extensive experiments are conducted on these ToD tasks in realistic few-shot learning scenarios with small validation data. Empirical results demonstrate that the proposed CINS approach consistently improves techniques that finetune PLMs with raw input or short prompts.

Figures (5)

• Figure 1: The unified framework of applying Comprehensive Instruction to different ToD downstream tasks. For each task in a row, the input is concatenated with the customized instruction (definition, constraint, prompt) before feeding to a T5 model to generate different types of output. An example instruction for NLU (intent classification) is given in the upper dashed box, and more details for different tasks are elaborated in Table \ref{['table:cp_def']}.
• Figure 2: Formulation comparison for Standard input, Prompt Engineering, and Comprehensive Instruction.
• Figure 3: Results of IC (1-shot), DST (1% data), NLG (1-shot) with different types of prompts with T5-small. Different prompts tested for IC and DST are provided in Table \ref{['table:prompt_type']}. For the same prompt backbone, the version with (D) stands for a Declarative expression, while the version with (Q) stands for a Question expression. Standard deviations over different random seeds are also plotted.
• Figure 4: Accuracy improvements of CIns over STD/PE in different configurations of intent classification (orange) and dialog state tracking (blue) tasks.
• Figure 5: Accuracy with standard deviation of CIns in different domains with varying number (5,10,15,20) of validation data per intent label for the 5-shot intent classification task. Validation data are only used for early-stopping T5-small model training. Validation size 5 is the same as the 5-shot training data.