SphOR: A Representation Learning Perspective on Open-set Recognition for Identifying Unknown Classes in Deep Learning Models

TL;DR

This work tackles open-set recognition by reframing feature learning on a hypersphere and modeling each class with von Mises–Fisher prototypes. The SphOR framework employs soft class assignments, a generalized vMF-based loss with Mixup-enabled soft labels, and a prototype-repulsion term to achieve strong intra-class compactness and inter-class separability, while using a linear classifier and adaptive rejection thresholds for unknown detection. Across diverse OSR benchmarks, SphOR delivers state-of-the-art results, including superior unknown and near/far-OOD detection, while maintaining competitive closed-set accuracy. Ablation studies and extended analyses demonstrate that Mixup and label smoothing substantially improve semantic discriminability and uncertainty estimation, and highlight the method’s robustness and practical impact. Limitations include a single fixed prototype per class, suggesting future work on multiple prototypes with dynamic concentrations to better capture complex class structure.

Abstract

The widespread use of deep learning classifiers necessitates Open-set recognition (OSR), which enables the identification of input data not only from classes known during training but also from unknown classes that might be present in test data. Many existing OSR methods are computationally expensive due to the reliance on complex generative models or suffer from high training costs. We investigate OSR from a representation-learning perspective, specifically through spherical embeddings. We introduce SphOR, a computationally efficient representation learning method that models the feature space as a mixture of von Mises-Fisher distributions. This approach enables the use of semantically ambiguous samples during training, to improve the detection of samples from unknown classes. We further explore the relationship between OSR performance and key representation learning properties which influence how well features are structured in high-dimensional space. Extensive experiments on multiple OSR benchmarks demonstrate the effectiveness of our method, producing state-of-the-art results, with improvements up-to 6% that validate its performance. Code at https://github.com/nadarasarbahavan/SpHOR

Figures (13)

• Figure 1: Our method promotes intra-class compactness and inter-class separability through a soft assignment mechanism, where samples are softly associated with their respective class prototypes based on their label information, rather than being strictly assigned. Additionally, we generate semantically vague samples using Mix-up during training. Concurrently, the method ensures that prototypes remain sufficiently distinct by repelling them from one another.
• Figure 2: Illustration of the effect of optimizing only the von Mises-Fisher distribution for intra-class compactness (a), where class prototypes may become too close, leading to potential misclassification of unknown class samples. By adding an additional loss term to optimize prototype-prototype repulsion (b), we enhance inter-class separability, reducing the risk of misclassifying ambiguous unknown class samples. The effect shown is the playtpus class which shares features with the beaver and duck class.
• Figure 3: Marco F1 in the CIFAR-10 experiment for various rejection thresholds $\lambda$.
• Figure 4: Marco F1 in the MNIST experiment for various rejection thresholds $\lambda$.
• Figure 5: CIFAR-10 as Known, SVHN as Unknown, No Mixup + Label Smoothing.