Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NECO: NEural Collapse Based Out-of-distribution detection

Mouïn Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, Gianni Franchi

TL;DR

NECO is introduced, a novel post-hoc method for OOD detection, which leverages the geometric properties of ``neural collapse'' and of principal component spaces to identify OOD data.

Abstract

Detecting out-of-distribution (OOD) data is a critical challenge in machine learning due to model overconfidence, often without awareness of their epistemological limits. We hypothesize that ``neural collapse'', a phenomenon affecting in-distribution data for models trained beyond loss convergence, also influences OOD data. To benefit from this interplay, we introduce NECO, a novel post-hoc method for OOD detection, which leverages the geometric properties of ``neural collapse'' and of principal component spaces to identify OOD data. Our extensive experiments demonstrate that NECO achieves state-of-the-art results on both small and large-scale OOD detection tasks while exhibiting strong generalization capabilities across different network architectures. Furthermore, we provide a theoretical explanation for the effectiveness of our method in OOD detection. Code is available at https://gitlab.com/drti/neco

NECO: NEural Collapse Based Out-of-distribution detection

TL;DR

NECO is introduced, a novel post-hoc method for OOD detection, which leverages the geometric properties of ``neural collapse'' and of principal component spaces to identify OOD data.

Abstract

Detecting out-of-distribution (OOD) data is a critical challenge in machine learning due to model overconfidence, often without awareness of their epistemological limits. We hypothesize that ``neural collapse'', a phenomenon affecting in-distribution data for models trained beyond loss convergence, also influences OOD data. To benefit from this interplay, we introduce NECO, a novel post-hoc method for OOD detection, which leverages the geometric properties of ``neural collapse'' and of principal component spaces to identify OOD data. Our extensive experiments demonstrate that NECO achieves state-of-the-art results on both small and large-scale OOD detection tasks while exhibiting strong generalization capabilities across different network architectures. Furthermore, we provide a theoretical explanation for the effectiveness of our method in OOD detection. Code is available at https://gitlab.com/drti/neco
Paper Structure (48 sections, 3 theorems, 8 equations, 17 figures, 8 tables, 1 algorithm)

This paper contains 48 sections, 3 theorems, 8 equations, 17 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Neural Collapse
  4. OOD Detection
  5. Preliminaries
  6. background and hypotheses
  7. neural collapse
  8. Out of Distribution Neural Collapse
  9. Neural collapse in the presence of OOD data.
  10. NEural Collapse based Out of distribution detection (NECO) method
  11. Theoretical justification
  12. Experiments & Results
  13. OOD Datasets.
  14. Evaluation Metrics.
  15. Experiment Details.
  16. ...and 33 more sections

Key Result

Theorem 4.1

We consider two datasets living in $\mathbb{R}^{D}$, $\{ D_{\text{OOD}}, D_{\tau} \}$ and a DNN $f_{\boldsymbol{\mathbf{\omega}}}(\cdot)=(g_{\boldsymbol{\mathbf{\omega}}}\circ h_{\boldsymbol{\mathbf{\omega}}})(\cdot)$ that satisfy NC1, NC2 and NC5. There $\exists ~d \ll D$ for PCA on $D_{\tau}$ s.t.

Figures (17)

  • Figure 1: Convergence to ID/OOD orthogonality for ViT-B (left), Resnet-18 (right) both trained on CIFAR-10 as ID and tested in the presence of OOD data. Dashed purple lines indicate the end of warm-up steps in the case of ViT and learning rate decay epochs for ResNet-18.
  • Figure 2: Feature projections on the first 2 principal components of a PCA fitted on CIFAR-10 (ID) using ViT penultimate layer representation. OOD data are ImageNet-O (left), Textures (middle), and SVHN (right). The Figure shows how NC1 (\ref{['NC_equation1']}) property is satisfied by ID data, and that OOD data lie around the origin.
  • Figure C.3: Example images from ImageNet-1k considered OOD datasets. Each row representing an OOD dataset: ImageNet-O, Textures, iNaturalist, SUN and Places365 respectively from top to bottom.
  • Figure C.4: Comparison of Performance metrics --- AUROC (left) and FPR95 (right) --- against the principal space dimension, for ViT (Top) and SwinV2 (Bottom), with ImageNet as ID data and different OOD datasets: iNaturalist, ImageNet-O, Textures, SUN, Places365.
  • Figure C.5: Comparison of Performance metrics --- AUROC (left) and FPR95 (right) --- against the principal space dimension, for ViT (Top) and ResNet-18 (Bottom), with CIFAR-10 as ID data, and different OOD datasets: SVHN, CIFAR-100.
  • ...and 12 more figures

Theorems & Definitions (6)

  • Theorem 4.1: NC1+NC2+NC5 imply NECO
  • Remark 4.1
  • Lemma A.1: Orthogonality conservation
  • proof
  • Theorem A.2: NC1+NC2+NC5 imply NECO
  • proof