Evaluating Anomaly Detectors for Simulated Highly Imbalanced Industrial Classification Problems

TL;DR

The paper investigates anomaly detectors under extreme class imbalance using a problem-agnostic TvS synthetic dataset in 2D and 10D, with training sizes from $1{,}000$ to $10{,}000$ and anomaly rates from $0.05\%$ to $20\%$, assessing generalization with a large test set of $40{,}000$ samples. It benchmarks 14 detectors across unsupervised, semi-supervised, and supervised categories, finding that unsupervised methods dominate when faulty examples are scarce ($<20$), while semi-supervised and supervised detectors improve markedly once about $30$–$50$ faulty examples are available, particularly as feature dimensionality grows. The study emphasizes that the total number of faulty training examples is a key determinant of detector performance and generalization, with notable variability across dataset size and dimensionality, and highlights risks of relying solely on validation metrics for detector selection. The results offer actionable guidance for deploying anomaly detection in industrial environments, including careful consideration of data composition, feature count, and generalization behavior. Future work could extend the simulation framework to more problem-specific detectors and incorporate model-based simulations to tailor evaluations to particular industrial contexts.

Abstract

Machine learning offers potential solutions to current issues in industrial systems in areas such as quality control and predictive maintenance, but also faces unique barriers in industrial applications. An ongoing challenge is extreme class imbalance, primarily due to the limited availability of faulty data during training. This paper presents a comprehensive evaluation of anomaly detection algorithms using a problem-agnostic simulated dataset that reflects real-world engineering constraints. Using a synthetic dataset with a hyper-spherical based anomaly distribution in 2D and 10D, we benchmark 14 detectors across training datasets with anomaly rates between 0.05% and 20% and training sizes between 1 000 and 10 000 (with a testing dataset size of 40 000) to assess performance and generalization error. Our findings reveal that the best detector is highly dependant on the total number of faulty examples in the training dataset, with additional healthy examples offering insignificant benefits in most cases. With less than 20 faulty examples, unsupervised methods (kNN/LOF) dominate; but around 30-50 faulty examples, semi-supervised (XGBOD) and supervised (SVM/CatBoost) detectors, we see large performance increases. While semi-supervised methods do not show significant benefits with only two features, the improvements are evident at ten features. The study highlights the performance drop on generalization of anomaly detection methods on smaller datasets, and provides practical insights for deploying anomaly detection in industrial environments.

Figures (14)

• Figure 1: Examples cases of structures seen in research.
• Figure 2: Example where false positives may be prioritized as they result in direct waste, whereas false negatives can be caught by a secondary inspection.
• Figure 3: Example where false negatives may be prioritized, as product directly exits system after passing screening.
• Figure 4: Examples of TvS distribution wheat_bayes_2025.
• Figure 5: Example of class distributions and scoring distributions for one TvS scenario.