Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection

TL;DR

The likelihood of variational DGMs is investigated and its detection performance could be improved in two directions: i) alleviating latent distribution mismatch, and ii) calibrating the dataset entropy-mutual integration.

Abstract

Unsupervised out-of-distribution (U-OOD) detection is to identify OOD data samples with a detector trained solely on unlabeled in-distribution (ID) data. The likelihood function estimated by a deep generative model (DGM) could be a natural detector, but its performance is limited in some popular "hard" benchmarks, such as FashionMNIST (ID) vs. MNIST (OOD). Recent studies have developed various detectors based on DGMs to move beyond likelihood. However, despite their success on "hard" benchmarks, most of them struggle to consistently surpass or match the performance of likelihood on some "non-hard" cases, such as SVHN (ID) vs. CIFAR10 (OOD) where likelihood could be a nearly perfect detector. Therefore, we appeal for more attention to incremental effectiveness on likelihood, i.e., whether a method could always surpass or at least match the performance of likelihood in U-OOD detection. We first investigate the likelihood of variational DGMs and find its detection performance could be improved in two directions: i) alleviating latent distribution mismatch, and ii) calibrating the dataset entropy-mutual integration. Then, we apply two techniques for each direction, specifically post-hoc prior and dataset entropy-mutual calibration. The final method, named Resultant, combines these two directions for better incremental effectiveness compared to either technique alone. Experimental results demonstrate that the Resultant could be a new state-of-the-art U-OOD detector while maintaining incremental effectiveness on likelihood in a wide range of tasks.

Figures (8)

• Figure 1: Illustration of the insights of the proposed methods. (a): The t-SNE visualization of latent representations on FashionMNIST(ID)/MNIST(OOD) dataset pair. (b) Visualization of the relationship between the number of singular values and the reconstruction error.
• Figure 2: Density plots and ROC curves. (a): Using the likelihood of a VAE trained on FashionMNIST leads to overestimation when detecting MNIST as OOD data, resulting in limited detection performance; (b): PHP could improve the detection performance; (c): SOTA method $\mathcal{LLR}^{ada}$ could degenerate the performance of likelihood; (d): PHP method satisfy the incremental effectiveness.
• Figure 3: (a) and (b) are respectively the visualizations of the scaled calculated entropy-mutual calibration ${\mathcal{C}}(\boldsymbol{x} )$ of CIFAR-10 (ID) and other OOD datasets, where the ${\mathcal{C}}(\boldsymbol{x} )$ of CIFAR-10 (ID) could achieve generally higher values. (c) is the OOD detection performance of dataset entropy-mutual calibration with different $n_{{id}}$ settings, which consistently outperforms likelihood.
• Figure 4: An illustration showcasing the difference between supervised and U-OOD detection.
• Figure 5: (a-d): Visualization of modeling a single-modal data distribution with a linear VAE; (e-h): Visualization of modeling a multi-modal data distribution with a linear VAE.

Theorems & Definitions (1)

• Definition 1: Incremental effectiveness on likelihood for U-OOD Detection