Look everywhere effects in anomaly detection

TL;DR

The paper tackles the look everywhere effect in ML-based anomaly detection by framing the problem within Classification Without Labels and comparing overdensity estimators based on binning or ML classifiers. It systematically studies how data usage strategies—training and testing on the same data, splitting data into halves, or employing $k$-fold cross-validation—affect $p$-value calibration and anomaly sensitivity using a 2d Gaussian toy model and the LHCO collider dataset. The key finding is that evaluating on training data yields strongly miscalibrated $p$-values due to overfitting to fluctuations, while independent testing calibrates well but reduces sensitivity; $k$-folding offers a practical balance between calibration and discovery potential, with early stopping providing classifier-dependent improvements. The results highlight a fundamental trade-off between p-value calibration and anomaly-sensitivity in ML AD frameworks and underscore the need for careful calibration in real-data analyses, potentially generalizing to other background estimation methods; code is released for reproducibility.

Abstract

Machine learning-based anomaly detection methods are able to search high-dimensional spaces for hints of new physics with much less theory bias than traditional searches. However, by searching in many directions all at once, the statistical power of these search strategies is diluted by a variant of the look elsewhere effect. We examine this challenge in detail, focusing on weakly supervised methods. We find that training and testing on the same data results in badly miscalibrated $p$-values due to the anomaly detector searching everywhere in the data and overfitting on statistical fluctuations. However, if these $p$-values can be calibrated, they may offer the best sensitivity to anomalies, since this approach uses all of the data. Conversely, training on half of the data and testing on the other half results in perfectly calibrated $p$-values, but at the cost of reduced sensitivity to anomalies. Similarly, regularization methods such as early stopping can help with $p$-value calibration but also possibly at the expense of sensitivity. Finally, we find that k-folding strikes an effective balance between calibration and sensitivity. Our findings are supported by numerical studies with Gaussian random variables as well as from collider physics using the LHC Olympics benchmark anomaly detection dataset.

Figures (5)

• Figure 1: Calibration curves showing empirical cumulative probability and calculated $p$-values observed in the binned classifier (top), the NN with and without early stopping (middle) and BDT with and without early stopping (bottom) for an evaluation on the training set (left), on the test set (middle) and using $k$-folding (right) using the 2d Gaussian toy data.
• Figure 2: Illustration of the miscalibration seen in k-fold cross validation. The grey-scale represents the excess in each bin.
• Figure 3: Calibration curves and fits showing empirical cumulative probability and calculated $p$-values observed for the NN with early stopping (top), NN without early stopping (middle) and for the BDT (bottom) for an evaluation on the training set (left), on the test set (middle) and using $k$-folding (right) on LHCO data.
• Figure 4: Maximum SIC as a function of the signal injection for classifiers trained on the statistics retained when evaluating on the training set, on the test set and using $k$-folding for NN with early stopping (left), without early stopping (middle) and for the BDT with early stopping (right).
• Figure 5: Uncorrected (dashed lines) and corrected (solid lines) $p$-values for different signal injections for NN with early stopping (top), without early stopping (middle) and for the BDT with early stopping (bottom) evaluated on the training set, on the test set and using $k$-folding at three different working points (left, middle, right).