Out-of-Distribution Detection using Neural Activation Prior

TL;DR

The paper introduces Neural Activation Prior (NAP), a simple, plug-and-play prior for out-of-distribution (OOD) detection that leverages within-channel activation distributions in the penultimate CNN layer before global pooling. The core idea is to compare, per channel $j$, the ratio $\frac{\max(\mathbf{A}_j)}{\text{Mean}(\mathbf{A}_j) + \epsilon}$ and aggregate across channels to form $S_{\text{NAP}}(x;f)$, with a Transformer analogue using CLS-token attention, $S_{\text{NAP}}^{\text{Former}} = \text{Max}(A)$. The method is orthogonal to existing post-pooling priors and is designed to be combined with scoring functions like the Energy score via $S_{\text{NAP-E}}(x;f,w) = -E(x;f)^w \cdot S_{\text{NAP}}(x;f)^{1-w}$, with $w$ chosen through pseudo-OOD data transformations. Empirically, NAP achieves state-of-the-art or strong results on CIFAR-10/100 and ImageNet benchmarks, extends effectively to Vision Transformers, and preserves in-distribution accuracy while improving OOD detection. This work suggests that rich pre-pooling activation statistics contain transferable OOD signals and can broadly enhance reliability in real-world applications.

Abstract

Out-of-distribution detection (OOD) is a crucial technique for deploying machine learning models in the real world to handle the unseen scenarios. In this paper, we first propose a simple yet effective Neural Activation Prior (NAP) for OOD detection. Our neural activation prior is based on a key observation that, for a channel before the global pooling layer of a fully trained neural network, the probability of a few neurons being activated with a large response by an in-distribution (ID) sample is significantly higher than that by an OOD sample. An intuitive explanation is that for a model fully trained on ID dataset, each channel would play a role in detecting a certain pattern in the ID dataset, and a few neurons can be activated with a large response when the pattern is detected in an input sample. Then, a new scoring function based on this prior is proposed to highlight the role of these strongly activated neurons in OOD detection. Our approach is plug-and-play and does not lead to any performance degradation on ID data classification and requires no extra training or statistics from training or external datasets. Notice that previous methods primarily rely on post-global-pooling features of the neural networks, while the within-channel distribution information we leverage would be discarded by the global pooling operator. Consequently, our method is orthogonal to existing approaches and can be effectively combined with them in various applications. Experimental results show that our method achieves the state-of-the-art performance on CIFAR benchmark and ImageNet dataset, which demonstrates the power of the proposed prior. Finally, we extend our method to Transformers and the experimental findings indicate that NAP can also significantly enhance the performance of OOD detection on Transformers, thereby demonstrating the broad applicability of this prior knowledge.

Figures (10)

• Figure 1: Global pooling leads to the neglect of distributional characteristics of activation values within channels, thereby making it difficult to distinguish between ID and OOD samples. (a) illustrates the activation value maps in the penultimate layer for in-distribution (ID) samples (left) and out-of-distribution (OOD) samples (right) within the same channel. In this layer, different channels typically focus on different semantic features. When a specific feature appears in an image, such as the central region in the left image, that location will have very high activation values. Although OOD samples lack these specific features, the model's unfamiliarity with OOD data leads to unpredictable activations, possibly causing weak noise activations (right image). This is detailed in sun2021react. Existing methods often rely on pooled activation values for OOD detection. Thus, in (a), both ID and OOD samples have a mean channel activation of 1.15, making them indistinguishable by existing methods. However, the NAP score proposed in this paper can effectively distinguish between them. More examples like (a) are provided in the Appendix \ref{['appendix:vis']}. (b) shows the activation distribution differences between ID and OOD data within the channel. The horizontal axis represents the average activation value, while the vertical axis represents the maximum activation value. Interestingly, at the same average activation level, ID data (CIFAR-10) shows significantly higher maximum activation values than OOD data (textures). This pattern is not unique to the $333^\text{rd}$ channel but is observable in most channels, corroborating the phenomenon in (a).
• Figure 2: Score distribution visualization using DenseNet on CIFAR-10 (ID) and iSun (OOD) The integration of (a) Energy Score and (b) NAP Score through multiplication yields the (c) Energy $\times$ NAP Score, demonstrating superior differentiation between ID and OOD datasets. The effectiveness of this approach is attributed to the orthogonal nature of the proposed NAP relative to conventional OOD detection methods exemplified by the Energy Score. This illustrates that a simple multiplicative combination with NAP enhances detection capability. Importantly, the objective is not merely to surpass the performance of the Energy Score itself, but to underscore the synergistic potential of NAP as a complementary enhancement to the Energy Score and similar method.
• Figure 3: Illustration identifying the focus zone of Neural Activation Prior (NAP) in classification neural networks. The figure highlights the specific position of NAP in the network: (a) For the CNN-based model, the activation value in the green part is the location of the NAP proposed in this article, which is before global pooling. (b) For the Transformer model, NAP is highlighted within the cls token's attention weights in the last Transformer block, illustrating a targeted approach in contrast to most existing methods that focus on regions after global pooling.
• Figure 4: Activation distribution at different positions within the DenseNet architecture huang2017densely applied to CIFAR-10 krizhevsky2009cifar and Places365 datasets zhou2017places. For this analysis, four specific locations within the network were chosen: (a) after the first convolution layer, (b) just before the pooling operation in the first transition block, (c) just before the pooling operation in the second transition block, and (d) right before the final global pooling layer. It is observed that with increasing depth, the separability between ID and OOD samples becomes more pronounced in the two-dimensional space defined by the maximum and average activation values.
• Figure 5: Activation distribution at different positions within the DenseNet architecture huang2017densely applied to CIFAR-10 and Places365 datasets. For this analysis, four specific locations within the network were chosen: (a) after the first convolution layer, (b) just before the pooling operation in the first transition block, (c) just before the pooling operation in the second transition block, and (d) right before the final global pooling layer. Note that we only include data points with an average activation over $0.1$. As shown in the figure, the first three selected layers show a less marked distinction between ID and OOD samples, while the fourth layer, preceding the final global pooling layer, demonstrates clearer separability and enhanced stability. Therefore, the fourth selected layer (the penultimate layer) is more suitable for developing a scoring function for OOD detection.