Revisiting Energy-Based Model for Out-of-Distribution Detection

TL;DR

This work tackles the challenge of robust out-of-distribution (OOD) detection without relying on curated real outliers by introducing peripheral-distribution (PD) data generated via simple transformations. Grounded in energy-based models (EBMs), it defines an energy barrier that separates in-distribution (ID) from PD and OOD samples, and proposes an energy-barrier loss (OEST*) to enforce this separation while maintaining ID accuracy. The key contributions are the PD data concept, the energy-barrier theory with a principled loss, and extensive experiments showing competitive to state-of-the-art OOD detection on CIFAR-10/100 across near and far OOD benchmarks, with improved generalization and backbone-insensitive performance. The approach offers a practical, scalable alternative to real outliers, potentially enabling more robust open-world perception in applications like autonomous systems and medical imaging, where curated OOD data are scarce.

Abstract

Out-of-distribution (OOD) detection is an essential approach to robustifying deep learning models, enabling them to identify inputs that fall outside of their trained distribution. Existing OOD detection methods usually depend on crafted data, such as specific outlier datasets or elaborate data augmentations. While this is reasonable, the frequent mismatch between crafted data and OOD data limits model robustness and generalizability. In response to this issue, we introduce Outlier Exposure by Simple Transformations (OEST), a framework that enhances OOD detection by leveraging "peripheral-distribution" (PD) data. Specifically, PD data are samples generated through simple data transformations, thus providing an efficient alternative to manually curated outliers. We adopt energy-based models (EBMs) to study PD data. We recognize the "energy barrier" in OOD detection, which characterizes the energy difference between in-distribution (ID) and OOD samples and eases detection. PD data are introduced to establish the energy barrier during training. Furthermore, this energy barrier concept motivates a theoretically grounded energy-barrier loss to replace the classical energy-bounded loss, leading to an improved paradigm, OEST*, which achieves a more effective and theoretically sound separation between ID and OOD samples. We perform empirical validation of our proposal, and extensive experiments across various benchmarks demonstrate that OEST* achieves better or similar accuracy compared with state-of-the-art methods.

Figures (4)

• Figure 1: (a) The t-SNE visualization of representations from CIFAR-10 (Red) test samples, rotated CIFAR-10 (Orange) test samples, CIFAR-100 (Blue), SVHN (Green) and ImageNet (Purple) before and after applying our training strategy OEST. Specifically, the embedding features are extracted from the penultimate layer of ResNet-18 classifier (trained on CIFAR-10). (b) We illustrate the feature space as a series of concentric spherical shells, where each shell corresponds to a certain energy level. The innermost shell contains in-distribution samples with the lowest energy, represented by $\bm{x}$. Moving outward, the orange points indicate augmented peripheral-distribution samples, denoted by $\bm{x}^{+}$. OEST establishes an energy barrier (reading $\gamma_\alpha$) between ID ($\bm{x}$) and PD ($\bm x^+$) data, thus separating ID and out-of-distribution samples ($\bm{x}'$).
• Figure 2: Visualization of the original image and the considered simple transformations. The difference between our design and a baseline method CSItack2020csi is also exhibited. CSI must select different suitable transformations elaborately for each particular scenario, and specifically for CIFAR-10 CSI chooses rotation. However, our method utilizes all kinds of transformations.
• Figure 3: Visualization of $Z$ during the tuning process of OEST on CIFAR-10. Here, $Z$ is computed as the empirical aggregation of all images used in zhang2023openood.
• Figure 4: OOD detection performance of a ResNet-18 classifier trained on CIFAR-10 as the in-distribution dataset, evaluated under varying values of hyperparameters $\alpha$ and $\beta$. In (a), $\beta$ is fixed at 10, and the effect of changing $\alpha$ is shown. In (b), $\alpha$ is fixed at 0.2, and the impact of changing $\beta$ is shown. Higher AUROC and Accuracy for both experiments indicate better performance, while lower FPR95 reflects better performance.

Theorems & Definitions (5)

• Remark
• Theorem 1
• Remark
• Theorem 1
• proof