DROID: Dual Representation for Out-of-Scope Intent Detection

TL;DR

DROID tackles out-of-scope (OOS) intent detection in task-oriented dialogue by fusing two complementary sentence encoders—a general-purpose Universal Sentence Encoder (USE) and a domain-adapted TSDAE—within a lightweight end-to-end framework. A single calibrated threshold on in-domain validation differentiates known intents from OOS without post-hoc scoring or strong distributional assumptions, while synthetic feature-space outliers and open-domain negatives strengthen boundary learning. With only 1.56M trainable parameters, DROID achieves state-of-the-art macro-F1 on Known and Unknown intents across CLINC-150, BANKING77, and STACKOVERFLOW, demonstrating robustness in low-resource settings and under varying known-intent coverage. The approach offers an efficient, deployable alternative to large LLM-based baselines, highlighting the benefits of dual representations and explicit confidence calibration for open-world intent understanding in real-time systems.

Abstract

Detecting out-of-scope (OOS) user utterances remains a key challenge in task-oriented dialogue systems and, more broadly, in open-set intent recognition. Existing approaches often depend on strong distributional assumptions or auxiliary calibration modules. We present DROID (Dual Representation for Out-of-Scope Intent Detection), a compact end-to-end framework that combines two complementary encoders -- the Universal Sentence Encoder (USE) for broad semantic generalization and a domain-adapted Transformer-based Denoising Autoencoder (TSDAE) for domain-specific contextual distinctions. Their fused representations are processed by a lightweight branched classifier with a single calibrated threshold that separates in-domain and OOS intents without post-hoc scoring. To enhance boundary learning under limited supervision, DROID incorporates both synthetic and open-domain outlier augmentation. Despite using only 1.5M trainable parameters, DROID consistently outperforms recent state-of-the-art baselines across multiple intent benchmarks, achieving macro-F1 improvements of 6--15% for known and 8--20% for OOS intents, with the most significant gains in low-resource settings. These results demonstrate that dual-encoder representations with simple calibration can yield robust, scalable, and reliable OOS detection for neural dialogue systems.

Figures (6)

• Figure 1: End-to-end pipeline of DROID. Stage 1: unsupervised domain adaptation of the TSDAE encoder on in-domain unlabeled text. Stage 2: supervised $(K{+}1)$ training with two frozen encoders (USE, domain-adapted TSDAE); per-branch projections produce embeddings for known intents, synthetic feature-space outliers (convex mixes of distinct known-class embeddings), and open-domain negatives. A light-weight MLP learns the $(K{+}1)$ classifier, and a single threshold $T$ is calibrated on in-domain validation data. Stage 3: inference on test utterances using the thresholded decision rule to separate known vs. OOS.
• Figure 2: Head architecture of DROID. USE and TSDAE embeddings are passed through parallel 5-layer MLP branches (sizes $512$–$256$–$256$–$256$–$256$; ReLU; BatchNorm and Dropout $0.4$ after the first two layers) to 256-d projections. The projections are concatenated (512-d) and fed to a 3-layer classifier (sizes $128$–$128$–$128$; ReLU; Dropout $0.4$ after the first two layers) followed by a linear layer to $(K{+}1)$ logits and softmax. A single calibrated threshold on the maximum softmax decides OOS at inference.
• Figure 3: Macro F1 versus label ratio on CLINC-150, BANKING-77, and STACKOVERFLOW. Top: Known; Bottom: Unknown. Columns correspond to Known-intent coverage (25/50/75%). DROID (red) maintains high and balanced performance under limited supervision.
• Figure 4: Ablation on the thresholded decision rule (CLINC-150). Incorporating the calibrated threshold improves Unknown F1 at 25/50/75% coverage while maintaining Known F1.
• Figure 5: Effect of class weighting across label ratios and coverages. Weights are most impactful for Known F1 in low-label regimes; Unknown F1 remains strong with marginal gains from weighting.