Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Open-World Lottery Ticket Hypothesis for OOD Intent Classification

Yunhua Zhou, Pengyu Wang, Peiju Liu, Yuxin Wang, Xipeng Qiu

TL;DR

The paper addresses the challenge of out-of-domain (OOD) intent classification by identifying overparameterization as a driver of overconfidence and proposing calibrated subnetworks obtained via pruning. It introduces the Open-world Lottery Ticket Hypothesis (OLT), which finds a winning subnetwork through one-shot magnitude pruning that preserves in-domain (IND) accuracy while improving OOD detection, aided by temperature scaling. Empirical results on four real-world datasets show consistent improvements over strong baselines and demonstrate compatibility with multiple OOD scoring functions, highlighting the practical impact of principled calibration. The work offers a principled, scalable path to robust open-world intent understanding and lays groundwork for extending calibrated lottery tickets to other architectures and modalities, including potential applications beyond discriminative models.

Abstract

Most existing methods of Out-of-Domain (OOD) intent classification rely on extensive auxiliary OOD corpora or specific training paradigms. However, they are underdeveloped in the underlying principle that the models should have differentiated confidence in In- and Out-of-domain intent. In this work, we shed light on the fundamental cause of model overconfidence on OOD and demonstrate that calibrated subnetworks can be uncovered by pruning the overparameterized model. Calibrated confidence provided by the subnetwork can better distinguish In- and Out-of-domain, which can be a benefit for almost all post hoc methods. In addition to bringing fundamental insights, we also extend the Lottery Ticket Hypothesis to open-world scenarios. We conduct extensive experiments on four real-world datasets to demonstrate our approach can establish consistent improvements compared with a suite of competitive baselines.

The Open-World Lottery Ticket Hypothesis for OOD Intent Classification

TL;DR

The paper addresses the challenge of out-of-domain (OOD) intent classification by identifying overparameterization as a driver of overconfidence and proposing calibrated subnetworks obtained via pruning. It introduces the Open-world Lottery Ticket Hypothesis (OLT), which finds a winning subnetwork through one-shot magnitude pruning that preserves in-domain (IND) accuracy while improving OOD detection, aided by temperature scaling. Empirical results on four real-world datasets show consistent improvements over strong baselines and demonstrate compatibility with multiple OOD scoring functions, highlighting the practical impact of principled calibration. The work offers a principled, scalable path to robust open-world intent understanding and lays groundwork for extending calibrated lottery tickets to other architectures and modalities, including potential applications beyond discriminative models.

Abstract

Most existing methods of Out-of-Domain (OOD) intent classification rely on extensive auxiliary OOD corpora or specific training paradigms. However, they are underdeveloped in the underlying principle that the models should have differentiated confidence in In- and Out-of-domain intent. In this work, we shed light on the fundamental cause of model overconfidence on OOD and demonstrate that calibrated subnetworks can be uncovered by pruning the overparameterized model. Calibrated confidence provided by the subnetwork can better distinguish In- and Out-of-domain, which can be a benefit for almost all post hoc methods. In addition to bringing fundamental insights, we also extend the Lottery Ticket Hypothesis to open-world scenarios. We conduct extensive experiments on four real-world datasets to demonstrate our approach can establish consistent improvements compared with a suite of competitive baselines.
Paper Structure (19 sections, 1 theorem, 19 equations, 3 figures, 3 tables)

This paper contains 19 sections, 1 theorem, 19 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Problem Statement
  5. Lottery Tickets Less Overconfident
  6. The Road to Open-would Lottery Tickets
  7. OOD Detection with Lottery Tickets
  8. Experiments
  9. Datasets
  10. Evaluation Metrics and Baselines
  11. Experimental Setting
  12. Main Results
  13. Analysis and Discussions
  14. A Mask for OOD Intent Classification
  15. Towards Open-world Lottery Ticket
  16. ...and 4 more sections

Key Result

Theorem 3.1

Let $x_{A}\in D_{\text{IND}}$ and $x_{B}\in D_{\text{OOD}}$ be from IND and OOD respectively, the logits outputed by pre-trained model $\mathcal{F}$ are $\bm{\phi}_{A}=\{a_1,...,a_k\}$ and $\bm{\phi}_{B}=\{b_1,...,b_k\}$ respectively. Suppose $a_1=\max\phi_{A}$ and $b_1=\max\phi_{B}$ and the probabi

Figures (3)

  • Figure 1: Plots showing (Top) Reliability diagrams and (Bottom) The distribution of In-and Out-of-domain uncertainty scores in the Stackoverflow dataset. The OLT denotes our proposed Open-world Lottery Ticket. The reliability diagrams (pink) are about the function of confidence, which measures the gap (i.e., miscalibration) between expected sample accuracy (black) and confidence. The Maximum Calibration Error (MCE) measures the maximum gap. If a model meets perfect calibration, the gap is zero and the diagrams disappear.
  • Figure 2: Masked Network vs. Baselines (Clinc-Full). The red dotted line marks the performance of masked network (subnetwork). Left Y-axis represents the accuary and Right Y-axis represents the FNR@95%TPR (The lower the value, the better).
  • Figure 3: Effect of Temperature Scaling. As T becomes larger, the benefits brought by T will soon become smaller.

Theorems & Definitions (2)

  • Theorem 3.1
  • proof