Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection

Bo Liu, Liming Zhan, Yujie Feng, Zexin Lu, Chengqiang Xie, Lei Xue, Albert Y. S. Lam, Xiao-Ming Wu

TL;DR

This study presents a novel fine-tuning framework for large language models (LLMs) aimed at enhancing in-distribution (ID) intent classification and out-of-distribution (OOD) intent detection, which utilizes semantic matching with prototypes derived from ID class names.

Abstract

In the realm of task-oriented dialogue systems, a robust intent detection mechanism must effectively handle malformed utterances encountered in real-world scenarios. This study presents a novel fine-tuning framework for large language models (LLMs) aimed at enhancing in-distribution (ID) intent classification and out-of-distribution (OOD) intent detection, which utilizes semantic matching with prototypes derived from ID class names. By harnessing the highly distinguishable representations of LLMs, we construct semantic prototypes for each ID class using a diversity-grounded prompt tuning approach. We rigorously test our framework in a challenging OOD context, where ID and OOD classes are semantically close yet distinct, referred to as \emph{near} OOD detection. For a thorough assessment, we benchmark our method against the prevalent fine-tuning approaches. The experimental findings reveal that our method demonstrates superior performance in both few-shot ID intent classification and near-OOD intent detection tasks.

Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection

TL;DR

This study presents a novel fine-tuning framework for large language models (LLMs) aimed at enhancing in-distribution (ID) intent classification and out-of-distribution (OOD) intent detection, which utilizes semantic matching with prototypes derived from ID class names.

Abstract

In the realm of task-oriented dialogue systems, a robust intent detection mechanism must effectively handle malformed utterances encountered in real-world scenarios. This study presents a novel fine-tuning framework for large language models (LLMs) aimed at enhancing in-distribution (ID) intent classification and out-of-distribution (OOD) intent detection, which utilizes semantic matching with prototypes derived from ID class names. By harnessing the highly distinguishable representations of LLMs, we construct semantic prototypes for each ID class using a diversity-grounded prompt tuning approach. We rigorously test our framework in a challenging OOD context, where ID and OOD classes are semantically close yet distinct, referred to as \emph{near} OOD detection. For a thorough assessment, we benchmark our method against the prevalent fine-tuning approaches. The experimental findings reveal that our method demonstrates superior performance in both few-shot ID intent classification and near-OOD intent detection tasks.
Paper Structure (22 sections, 6 equations, 3 figures, 3 tables)

This paper contains 22 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Problem Statement
  4. Semantic Matching as ID Classification
  5. Diversity-grounded Channel Prototypical Learning
  6. Class Prototypes
  7. Diversity Learning
  8. Semantic Matching
  9. Optimization
  10. OOD Detection with Post-hoc Scoring
  11. Experiment
  12. Experimental Setup
  13. Datasets
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Our proposed semantic matching framework. We prompt class name into a sequence of learnable tokens and forward them into LLMs to generate class prototypes. With further training between prototypes and input representations via matching loss and diversity loss, better ID classification and OOD detection can be performed.
  • Figure 2: Performance of various class prototypes in the 5-shot scenario using CLINC-Banking dataset.
  • Figure 3: UMAP mcinnes2018umap visualization of representations of test set and OOD data from 5-shot CLINC-Bank. The purple means the OOD data. The star indicates the learned prototype of each class.