Can Dense Connectivity Benefit Outlier Detection? An Odyssey with NAS

TL;DR

The paper addresses the challenge of robust OOD detection by examining how CNN architectural design, specifically dense connectivity, impacts detection performance. It introduces Dense Connectivity Search of Outlier Detector (DCSOD), a predictor-based NAS framework that searches a flexible DAG-based cell space with dense operator connections, coupled with an evolving distillation strategy to stabilize OOD evaluation. Through extensive CIFAR experiments, DCSOD achieves state-of-the-art AUROC for near-OOD tasks and demonstrates that dense connectivity can significantly improve outlier detection when guided by stable evaluation and multi-view feature learning. The approach provides a practical pathway to design architecture-aware detectors with improved reliability and ranking during NAS.

Abstract

Recent advances in Out-of-Distribution (OOD) Detection is the driving force behind safe and reliable deployment of Convolutional Neural Networks (CNNs) in real world applications. However, existing studies focus on OOD detection through confidence score and deep generative model-based methods, without considering the impact of DNN structures, especially dense connectivity in architecture fabrications. In addition, existing outlier detection approaches exhibit high variance in generalization performance, lacking stability and confidence in evaluating and ranking different outlier detectors. In this work, we propose a novel paradigm, Dense Connectivity Search of Outlier Detector (DCSOD), that automatically explore the dense connectivity of CNN architectures on near-OOD detection task using Neural Architecture Search (NAS). We introduce a hierarchical search space containing versatile convolution operators and dense connectivity, allowing a flexible exploration of CNN architectures with diverse connectivity patterns. To improve the quality of evaluation on OOD detection during search, we propose evolving distillation based on our multi-view feature learning explanation. Evolving distillation stabilizes training for OOD detection evaluation, thus improves the quality of search. We thoroughly examine DCSOD on CIFAR benchmarks under OOD detection protocol. Experimental results show that DCSOD achieve remarkable performance over widely used architectures and previous NAS baselines. Notably, DCSOD achieves state-of-the-art (SOTA) performance on CIFAR benchmark, with AUROC improvement of $\sim$1.0%.

Figures (6)

• Figure 1: Classification Accuracy v.s. OOD AUROC of ResNets with different number of layers. Better accuracy does not indicate higher AUROC score.
• Figure 2: The dense connectivity search space, with three examples of dense connectivity in a cell. Standard structure such as ResNet and DenseNet are included.
• Figure 3: Evaluation variance of sampled networks from search space. Energy(S) and Energy(T) stands for energy scoring and training respectively.
• Figure 4: Comparison of out-of-distribution (OOD) AUROC and predictor Kendall's $\tau$ for different sample-iteration trade-offs.
• Figure 5: Visualization of the loss landscape for cross-entropy and outlier distillation loss.