Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ODIM: Outlier Detection via Likelihood of Under-Fitted Generative Models

Dongha Kim, Jaesung Hwang, Jongjin Lee, Kunwoong Kim, Yongdai Kim

TL;DR

The paper tackles unsupervised outlier detection by proposing ODIM, a method that exploits an inlier-memorization (IM) effect in under-fitted generative models to separate inliers from outliers using a limited number of updates. It leverages an IWAE-based objective, a targeted min-max preprocessing scheme, and an ensemble of models to produce robust outlier scores, claiming state-of-the-art performance across tabular, image, and text data while remaining computationally efficient. The authors provide theoretical intuition for the IM effect, validate it with empirical analyses, and demonstrate practical extensions to partially labeled data and differential privacy. The work offers a domain-agnostic, scalable approach to UOD with potential for real-time and privacy-preserving applications, supported by extensive experiments on nearly 60 datasets. The key contribution is a simple yet powerful framework that reconsiders likelihood as a discriminative signal when deep generative models are intentionally kept under-fitted during early training.

Abstract

The unsupervised outlier detection (UOD) problem refers to a task to identify inliers given training data which contain outliers as well as inliers, without any labeled information about inliers and outliers. It has been widely recognized that using fully-trained likelihood-based deep generative models (DGMs) often results in poor performance in distinguishing inliers from outliers. In this study, we claim that the likelihood itself could serve as powerful evidence for identifying inliers in UOD tasks, provided that DGMs are carefully under-fitted. Our approach begins with a novel observation called the inlier-memorization (IM) effect-when training a deep generative model with data including outliers, the model initially memorizes inliers before outliers. Based on this finding, we develop a new method called the outlier detection via the IM effect (ODIM). Remarkably, the ODIM requires only a few updates, making it computationally efficient-at least tens of times faster than other deep-learning-based algorithms. Also, the ODIM filters out outliers excellently, regardless of the data type, including tabular, image, and text data. To validate the superiority and efficiency of our method, we provide extensive empirical analyses on close to 60 datasets.

ODIM: Outlier Detection via Likelihood of Under-Fitted Generative Models

TL;DR

The paper tackles unsupervised outlier detection by proposing ODIM, a method that exploits an inlier-memorization (IM) effect in under-fitted generative models to separate inliers from outliers using a limited number of updates. It leverages an IWAE-based objective, a targeted min-max preprocessing scheme, and an ensemble of models to produce robust outlier scores, claiming state-of-the-art performance across tabular, image, and text data while remaining computationally efficient. The authors provide theoretical intuition for the IM effect, validate it with empirical analyses, and demonstrate practical extensions to partially labeled data and differential privacy. The work offers a domain-agnostic, scalable approach to UOD with potential for real-time and privacy-preserving applications, supported by extensive experiments on nearly 60 datasets. The key contribution is a simple yet powerful framework that reconsiders likelihood as a discriminative signal when deep generative models are intentionally kept under-fitted during early training.

Abstract

The unsupervised outlier detection (UOD) problem refers to a task to identify inliers given training data which contain outliers as well as inliers, without any labeled information about inliers and outliers. It has been widely recognized that using fully-trained likelihood-based deep generative models (DGMs) often results in poor performance in distinguishing inliers from outliers. In this study, we claim that the likelihood itself could serve as powerful evidence for identifying inliers in UOD tasks, provided that DGMs are carefully under-fitted. Our approach begins with a novel observation called the inlier-memorization (IM) effect-when training a deep generative model with data including outliers, the model initially memorizes inliers before outliers. Based on this finding, we develop a new method called the outlier detection via the IM effect (ODIM). Remarkably, the ODIM requires only a few updates, making it computationally efficient-at least tens of times faster than other deep-learning-based algorithms. Also, the ODIM filters out outliers excellently, regardless of the data type, including tabular, image, and text data. To validate the superiority and efficiency of our method, we provide extensive empirical analyses on close to 60 datasets.
Paper Structure (43 sections, 2 theorems, 34 equations, 5 figures, 28 tables, 2 algorithms)

This paper contains 43 sections, 2 theorems, 34 equations, 5 figures, 28 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Outlier detection
  3. Likelihood-based approaches in OD
  4. Overview of our method
  5. Related works
  6. Semi-supervised outlier detection
  7. Unsupervised outlier detection
  8. Proposed method
  9. Notations and definitions
  10. Motivation: inlier-memorization effect
  11. Theoretical analysis
  12. Choice of pre-processing technique
  13. Algorithm description
  14. Choice of the learning algorithm for a DGM
  15. Selection of the optimal number of updates
  16. ...and 28 more sections

Key Result

Proposition 3.1

Since the generative model $p({\bf x};\theta)$ is related only with the parameter $\theta$, we only consider the gradient with respect to $\theta$. For an input vector ${\bf x}$, the following holds: Proposition prop:1 indicates that in the early phases of learning, the magnitude of the gradient of the VAE is proportional to the $l_1$-norm of the input vector on average. This implies that when the

Figures (5)

  • Figure 1: An illustration of the ODIM method.
  • Figure 2: ( 1st to 5th) The distributions of the per-sample (normalized) VAE loss values of Cardio after 10, 20, 30, 40, and 500 training updates, respectively. For each panel, we depict the histograms of inliers and outliers separately. ( Last) The positive relationship between the Wasserstein distance and identifying performance (AUC) on Cardio.
  • Figure 3: Distributions of per-sample (Left) input $l_1$-norm values and (Right) gradient $l_2$-norm values of VAE loss on Cardio. We consider two pre-processing schemes to normalize each feature: 1) (Upper) min-max scaling, and 2) (Lower) standardization.
  • Figure 4: (From left to right) 1) AUC results on tabular datasets with various values of $K$. 2) AUC results on tabular datasets with various values of $N_{\text{pat}}$. 3) AUC results on tabular datasets with various values of $B$. 4) AUC results on FMNIST for each class with various learning rates. We vary the learning rate from 1e-4 to 1e-1.
  • Figure B.1: ( Left to Right) The distributions of the per-sample (normalized) negative log-likelihood values of GLOW on FMNIST after 10, 20, 30, and 40 updates, respectively. For each panel, we depict the histograms of inliers and outliers separately.

Theorems & Definitions (4)

  • Proposition 3.1
  • Remark 3.2
  • Remark 3.3
  • Proposition 3.4