Universal Novelty Detection Through Adaptive Contrastive Learning

TL;DR

The paper tackles the challenge of universality in novelty detection, requiring robust generalization across training and testing distribution shifts. It introduces UNODE, a framework that combines adaptive negative-pair generation (AutoAugOOD) with contrastive learning to create near-OOD samples that diversify representation learning. By jointly optimizing a contrastive loss and a cross-entropy objective and by constructing a unified anomaly score from similarity and binary OOD signals, UNODE achieves superior transferability and robustness across standard, corrupted, and mixed supervision setups, with notable AUROC gains and reduced variance. This approach offers a scalable, adaptable method for universal novelty detection across varied domains and task setups, including one-class, unlabeled multi-class, and labeled multi-class scenarios.

Abstract

Novelty detection is a critical task for deploying machine learning models in the open world. A crucial property of novelty detection methods is universality, which can be interpreted as generalization across various distributions of training or test data. More precisely, for novelty detection, distribution shifts may occur in the training set or the test set. Shifts in the training set refer to cases where we train a novelty detector on a new dataset and expect strong transferability. Conversely, distribution shifts in the test set indicate the methods' performance when the trained model encounters a shifted test sample. We experimentally show that existing methods falter in maintaining universality, which stems from their rigid inductive biases. Motivated by this, we aim for more generalized techniques that have more adaptable inductive biases. In this context, we leverage the fact that contrastive learning provides an efficient framework to easily switch and adapt to new inductive biases through the proper choice of augmentations in forming the negative pairs. We propose a novel probabilistic auto-negative pair generation method AutoAugOOD, along with contrastive learning, to yield a universal novelty detector method. Our experiments demonstrate the superiority of our method under different distribution shifts in various image benchmark datasets. Notably, our method emerges universality in the lens of adaptability to different setups of novelty detection, including one-class, unlabeled multi-class, and labeled multi-class settings. Code: https://github.com/mojtaba-nafez/UNODE

Figures (5)

• Figure 1: Evaluating Novelty Detection Performance: A Comparative Study on MVTecAD, CIFAR-10, and SVHN Datasets. Our proposed method, UNODE, consistently exhibits robust performance across all datasets, even under shifts in the testing sets. This highlights its superior transferability and generalizability in contrast to existing methods.
• Figure 2: An Examination of the AutoAugOOD Hard Augmentation Module: During the training process, our proposed module applies several hard augmentations to copies of the training set, creating multiple shifted distributions. The Kullback-Leibler (KL) divergence measures the ‘hardness’ of each augmentation by comparing the density of the original and shifted distributions. The final negative augmentation is a composite of several augmentations, weighted according to their respective KL divergence values.
• Figure 3: Impact of Test Set Distribution Shift on Error Rate: The model undergoes training with an auxiliary set comprising outliers distinct from inliers. During inference, a shift in the distribution of actual outliers leads to an increased error rate. An efficient choice of the auxiliary outlier dataset (with its mean closer to the inliers) can mitigate this issue to a certain degree.
• Figure 4: The training stage of the novelty detection framework incorporates Contrastive Loss and Cross Entropy Loss. Features are first extracted and then channeled through two concurrent pathways: one leading to a projection head for representation learning, and the other directly to the classification layer. This dual-pathway approach allows the model to categorize inputs as inlier or outlier, while simultaneously refining feature representations to improve detection score.
• Figure 5: 2D t-SNE visualizations display feature embeddings from a pre-trained backbone for the CIFAR-10 and MVTecAD datasets. These visualizations highlight the distributional shifts induced by specific augmentation techniques, i.e., Rotation and CutPaste. In the CIFAR-10 dataset, particularly within the car class, the Rotation augmentation forms distinct clusters, underscoring its capability in simulating outlier scenarios. Conversely, for the bottle class in MVTecAD, the CutPaste augmentation is notably effective, accurately mimicking the behavior of real outliers.