Multi-layer Radial Basis Function Networks for Out-of-distribution Detection

TL;DR

The paper introduces Multi-layer Radial Basis Function Networks (MLRBFNs) that integrate classification and out-of-distribution (OOD) detection within a single architecture. It addresses training challenges of deep RBF stacks by adding a depression mechanism to avoid 0-mapped classes and by decoupling predictions from confidence using binary cross-entropy, while leveraging k-means initializations to accelerate training. Through experiments on moons data, MNIST, and feature-extractor pipelines with foundation-model embeddings, MLRBFNs achieve competitive OOD detection performance against benchmarks like OpenOOD v1.5 and postprocessors, with robustness observed as the number of classification layers increases. The work demonstrates a promising direction for inherently distance-aware, OOD-sensitive networks and suggests future gains from deeper, more scalable RBF architectures and integration with other modern techniques. All mathematical notation is presented with proper Delimiters, and key equations reflect the distance-based activations and the depression mechanism that governs OOD behavior within the layers.

Abstract

Existing methods for out-of-distribution (OOD) detection use various techniques to produce a score, separate from classification, that determines how ``OOD'' an input is. Our insight is that OOD detection can be simplified by using a neural network architecture which can effectively merge classification and OOD detection into a single step. Radial basis function networks (RBFNs) inherently link classification confidence and OOD detection; however, these networks have lost popularity due to the difficult of training them in a multi-layer fashion. In this work, we develop a multi-layer radial basis function network (MLRBFN) which can be easily trained. To ensure that these networks are also effective for OOD detection, we develop a novel depression mechanism. We apply MLRBFNs as standalone classifiers and as heads on top of pretrained feature extractors, and find that they are competitive with commonly used methods for OOD detection. Our MLRBFN architecture demonstrates a promising new direction for OOD detection methods.

Figures (5)

• Figure 1: MLRBFNs are capable of correct classification with high confidence on the 4-class moons dataset. Away from the training data manifold, the network is not confident.
• Figure 2: Confidence of ID and OOD datasets when training an MLRBFN. In both subfigures, CIFAR-10 is the ID dataset, and CLIP ViT-B/16 is used to extract features. (a) shows the frequency of scores for the near-OOD dataset Tiny ImageNet, and (b) shows the scores for the far-OOD dataset Texture. The MLRBFN demonstrates strong separability between the ID and both OOD datasets.
• Figure 3: An MLRBFN trained on the 4-class moons dataset without the depression mechanism. Employing class-wise binary cross-entropy teaches the network to only predict one class with high confidence near each moon, but the network has very wide confidence regions near each moon. It is also very confident that inputs far away from all moons should be predicted as the blue class. Compare this to \ref{['fig:teaser']}, in which the depression mechanism produces confidence regions are restricted to around the data manifold.
• Figure 4: Confidences of an MLRBFN trained on the MNIST dataset. MNIST testing data confidence is blue, while OOD datasets FashionMNIST and KMNIST are in green and red, respectively. It is clear that MLRBFNs are capable of distinguishing between ID and OOD inputs throu
• Figure 5: AUROC for near- and far-OOD tasks on the CIFAR10 dataset. MSP was used for DNN OOD detection. As the number of classification layers increases, MLRBFNs maintain high AUROC for the OOD detection task, while DNNs have decreasing AUROC. This supports MLRBFNs as networks which are naturally capable of detecting OOD inputs.