Towards Unsupervised Validation of Anomaly-Detection Models
Lihi Idan
TL;DR
This work tackles unsupervised validation of anomaly-detection models, focusing on model selection and evaluation without labeled data. It introduces Accurately-Diverse ensembles that balance intra-ensemble agreement on general trends with intra-ensemble disagreement on exact rankings, operationalized through novel multi-way rank correlations and rank-cluster features. The Unsupervised Ensemble Divergence ($\mathcal{UED}$) score is proposed to compare a candidate model against the ensemble in a distance-based, weighted, unsupervised fashion. Empirical results on ten datasets show that Accurately-Diverse ensembles improve over the average unsupervised model and that $\mathcal{UED}$ correlates with supervised evaluation metrics, enabling effective unsupervised model selection and evaluation with practical impact for label-scarce regimes.
Abstract
Unsupervised validation of anomaly-detection models is a highly challenging task. While the common practices for model validation involve a labeled validation set, such validation sets cannot be constructed when the underlying datasets are unlabeled. The lack of robust and efficient unsupervised model-validation techniques presents an acute challenge in the implementation of automated anomaly-detection pipelines, especially when there exists no prior knowledge of the model's performance on similar datasets. This work presents a new paradigm to automated validation of anomaly-detection models, inspired by real-world, collaborative decision-making mechanisms. We focus on two commonly-used, unsupervised model-validation tasks -- model selection and model evaluation -- and provide extensive experimental results that demonstrate the accuracy and robustness of our approach on both tasks.