Mining In-distribution Attributes in Outliers for Out-of-distribution Detection

TL;DR

This paper addresses unreliable predictions in out-of-distribution (OOD) detection by revealing that OOD samples often contain in-distribution (ID) attributes. It introduces the Extended Multi-view Data Model (MVDM) and a MaxLogit-based OOD score to interpret ID attributes in outliers, along with MVOL, a multi-view learning objective that calibrates logits using auxiliary OOD data while respecting ID feature structure. Theoretical results show favorable bounds for both single-model and ensemble-distillation regimes, and empirical results demonstrate that MVOL outperforms strong baselines on CIFAR benchmarks and wild datasets, while preserving ID accuracy. This approach offers robust, calibrated OOD detection applicable even when auxiliary data are noisy or scarce, with practical implications for deploying reliable systems.

Abstract

Out-of-distribution (OOD) detection is indispensable for deploying reliable machine learning systems in real-world scenarios. Recent works, using auxiliary outliers in training, have shown good potential. However, they seldom concern the intrinsic correlations between in-distribution (ID) and OOD data. In this work, we discover an obvious correlation that OOD data usually possesses significant ID attributes. These attributes should be factored into the training process, rather than blindly suppressed as in previous approaches. Based on this insight, we propose a structured multi-view-based out-of-distribution detection learning (MVOL) framework, which facilitates rational handling of the intrinsic in-distribution attributes in outliers. We provide theoretical insights on the effectiveness of MVOL for OOD detection. Extensive experiments demonstrate the superiority of our framework to others. MVOL effectively utilizes both auxiliary OOD datasets and even wild datasets with noisy in-distribution data. Code is available at https://github.com/UESTC-nnLab/MVOL.

Figures (5)

• Figure 1: (a)-(c) Motivation and (d) overview of MVOL. (a) OOD data could have ID attributes, (b) outliers mainly have minor ID features, and (c) the model trained solely on ID data can produce over-activated logits for certain outliers. The gray-shaded area highlights those outliers with high probability. Refer to Appendix E for more experimental details.
• Figure 2: (a) Illustration of OOD data structure assumed in OE and our MVOL. (b) From our new assumption on out-of-distribution, OE blindly aligns the logits of all categories to the same level. Instead, MVOL adaptively aligns the logits of categories with associated minor ID features on outliers. The key insight is that a well-calibrated model should not display significantly distinct activation on these categories since minor ID features usually have small coefficients.
• Figure 3: MVOL and OE + MaxLogit with CIFAR-100 as ID dataset. (a) Logits of a training OOD sample, (b) ROC curve with Textures as the test OOD dataset.
• Figure 4: More examples for illustrating that OOD data could have ID attributes.
• Figure 5: More examples for illustrating the optimization objectives of OE and Our MVOL on auxiliary OOD samples. Blue bars represent our MVOL, and red bars represent OE. CIFAR-100 and 300K RandomImages provide ID and auxiliary OOD data, respectively.

Theorems & Definitions (20)

• Definition 1: data distributions $D_m^{in}$ and $D_s^{in}$
• Definition 2: $D^{in}$ and $Z^{in}$
• Definition 3: Out-of-distribution $D^{out}$
• Definition 4: $Z^{out}$
• Proposition 1
• Theorem 1: Calibrated Single Model
• Theorem 2: Calibrated Ensemble Distillation Model
• Definition 1: data distributions $D_m^{in}$ and $D_s^{in}$
• Definition 2: $D^{in}$ and $Z^{in}$
• Definition 3: Out-of-distribution $D^{out}$