Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vendi Novelty Scores for Out-of-Distribution Detection

Amey P. Pasarkar, Adji Bousso Dieng

TL;DR

The paper introduces the Vendi Novelty Score (VNS), a non-parametric, diversity-based OOD detector that leverages the Vendi Scores to quantify novelty in a class-conditional feature space. VNS computes a class-conditional novelty contribution for each class and aggregates these signals using the model’s predicted class probabilities over a top-K set, with an optional global density correction to capture dataset-wide novelty. Empirically, VNS achieves state-of-the-art or competitive OOD performance across CIFAR-10, CIFAR-100, and ImageNet-1K benchmarks with multiple architectures, while maintaining strong data-efficiency, working well with as little as 1% of the training data. The approach blends local (class-conditional) and global (dataset-wide) novelty signals in linear time and scales to large datasets, offering a practical and robust tool for safe deployment of deep models.

Abstract

Out-of-distribution (OOD) detection is critical for the safe deployment of machine learning systems. Existing post-hoc detectors typically rely on model confidence scores or likelihood estimates in feature space, often under restrictive distributional assumptions. In this work, we introduce a third paradigm and formulate OOD detection from a diversity perspective. We propose the Vendi Novelty Score (VNS), an OOD detector based on the Vendi Scores (VS), a family of similarity-based diversity metrics. VNS quantifies how much a test sample increases the VS of the in-distribution feature set, providing a principled notion of novelty that does not require density modeling. VNS is linear-time, non-parametric, and naturally combines class-conditional (local) and dataset-level (global) novelty signals. Across multiple image classification benchmarks and network architectures, VNS achieves state-of-the-art OOD detection performance. Remarkably, VNS retains this performance when computed using only 1% of the training data, enabling deployment in memory- or access-constrained settings.

Vendi Novelty Scores for Out-of-Distribution Detection

TL;DR

The paper introduces the Vendi Novelty Score (VNS), a non-parametric, diversity-based OOD detector that leverages the Vendi Scores to quantify novelty in a class-conditional feature space. VNS computes a class-conditional novelty contribution for each class and aggregates these signals using the model’s predicted class probabilities over a top-K set, with an optional global density correction to capture dataset-wide novelty. Empirically, VNS achieves state-of-the-art or competitive OOD performance across CIFAR-10, CIFAR-100, and ImageNet-1K benchmarks with multiple architectures, while maintaining strong data-efficiency, working well with as little as 1% of the training data. The approach blends local (class-conditional) and global (dataset-wide) novelty signals in linear time and scales to large datasets, offering a practical and robust tool for safe deployment of deep models.

Abstract

Out-of-distribution (OOD) detection is critical for the safe deployment of machine learning systems. Existing post-hoc detectors typically rely on model confidence scores or likelihood estimates in feature space, often under restrictive distributional assumptions. In this work, we introduce a third paradigm and formulate OOD detection from a diversity perspective. We propose the Vendi Novelty Score (VNS), an OOD detector based on the Vendi Scores (VS), a family of similarity-based diversity metrics. VNS quantifies how much a test sample increases the VS of the in-distribution feature set, providing a principled notion of novelty that does not require density modeling. VNS is linear-time, non-parametric, and naturally combines class-conditional (local) and dataset-level (global) novelty signals. Across multiple image classification benchmarks and network architectures, VNS achieves state-of-the-art OOD detection performance. Remarkably, VNS retains this performance when computed using only 1% of the training data, enabling deployment in memory- or access-constrained settings.
Paper Structure (26 sections, 1 theorem, 36 equations, 3 figures, 7 tables)

This paper contains 26 sections, 1 theorem, 36 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Vendi Novelty Score
  4. Experiments
  5. OOD Detection Performance Results
  6. Choices of Class-Conditional Novelty
  7. Choice of rank
  8. Data-efficient OOD Detection
  9. Conclusion
  10. Acknowledgements
  11. Additional Algorithm Details for VNS
  12. Scaling Novelty Scores to Account for Dataset Size
  13. Proof of Proposition \ref{['prop:lambda_max_update']}
  14. Comparison of Local and Global Novelty Scores
  15. Additional Experiment Details
  16. ...and 11 more sections

Key Result

Proposition 3.1

Let $\rho_{\text{global}} \in \mathbb{R}^{D\times D}$ be a symmetric positive semidefinite matrix with largest eigenvalue $\lambda_{\max}(\rho_{\text{global}})$ and corresponding unit-norm eigenvector $u_{\max}$. For a unit-norm vector $h(x)\in\mathbb{R}^D$ and dataset size $N$, define the rank-one Then the largest eigenvalue of $\rho_{\text{global}}'$ admits the first-order expansion Equivalent

Figures (3)

  • Figure 1: Conceptual overview of VNS. Left: Three in-distribution (ID) classes are shown as clusters in representation space (colored regions), along with a test out-of-distribution (OOD) sample (star). VNS computes a class-conditional novelty score for the OOD sample with respect to each class. Middle: Using only the class-conditional novelty score from the model's predicted class provides reasonable separation between ID and OOD samples. Right: Aggregating class-conditional novelty scores using prediction probabilities (Equation \ref{['eq:aggregate']}) yields improved ID-OOD separation, reducing the False Positive Rate (FPR).
  • Figure 2: Average eigenspectrum of class-conditional density matrices $\rho_c$ across models is dominated by the leading eigenvalue. The average values of the top-100 eigenvalues of $\rho_c$ are shown (log-scale $y$-axis), across three ImageNet-1K models.
  • Figure 3: VNS provides accurate OOD detection even with access to only $1\%$ of training data. FPR@95 is shown as a function of the percent of the training data used. Results across three ImageNet-1K models are displayed, averaged across five reproductions.

Theorems & Definitions (1)

  • Proposition 3.1: Accuracy of the Max Eigenvalue Update