Collective Outlier Detection and Enumeration with Conformalized Closed Testing

TL;DR

The paper tackles collective outlier detection under distribution-free guarantees by introducing ACODE, a framework that uses conformal inference to convert powerful, possibly black-box classifiers into principled conformity scores for global testing and enumeration. It automates the choice of classifier and two-sample testing procedure, integrating closed testing to yield simultaneous lower bounds on the number of outliers in any subset and a global outlier test, with data-driven tuning to maintain validity. The approach combines Shirashi’s locally most powerful rank tests and adaptive testing with exchangeability-based asymptotics, providing finite-sample validity and asymptotic power guarantees under mild assumptions. Empirical demonstrations on synthetic data and the LHCO particle-collision dataset show that ACODE achieves near-oracle performance in global detection and outlier enumeration, while remaining robust to selection bias and applicable to large-scale data. The work offers a practical, distribution-free toolkit for applications in finance, cybersecurity, and physics where collective anomalies are more detectable than individual outliers.

Abstract

This paper develops a flexible distribution-free method for collective outlier detection and enumeration, designed for situations in which the presence of outliers can be detected powerfully even though their precise identification may be challenging due to the sparsity, weakness, or elusiveness of their signals. This method builds upon recent developments in conformal inference and integrates classical ideas from other areas, including multiple testing, locally most powerful and adaptive rank tests, and non-parametric large-sample asymptotics. The key innovation lies in developing a principled and effective approach for automatically choosing the most appropriate machine learning classifier and two-sample testing procedure for a given data set. The performance of our method is investigated through extensive empirical demonstrations, including an analysis of the LHCO high-energy particle collision data set.

Figures (26)

• Figure 1: Preview of performance of ACODE on the LHCO data. Left: Median and 90th-quantile over repeated experiments for a 90% lower confidence bound for the number of outliers. Right: power against the global null hypothesis of no outliers at the 10% level (horizontal line). The results are shown as a function of the true number of outliers in a test set of cardinality 10,000. ACODE utilizes a testing procedure that may be adaptively selected (red curve) or fixed (other solid curves). Dotted curve: number of individual discoveries or power obtained by applying the Benjamini-Hochberg procedure (BH) to conformal $p$-values, controlling the false discovery rate below 10%.
• Figure 2: Median values for a 90% lower confidence bound on the number of outliers in a test set, computed by ACODE on synthetic data based on different classifiers and local testing procedures. The results are shown as a function of the true number of outliers within a test set of size 1000. The most adaptive version of ACODE can automatically select an effective classifier and local testing procedure in a data-driven way.
• Figure 3: Median values for a 90% lower confidence bound on the number of outliers within an adaptively selected subset of 1000 test points, in experiments similar to those of Figure \ref{['fig:exp-synthetic-1']}. The results are shown as a function of the proportion of selected test points and of the total number of outliers in the test set. The dashed curve corresponds to the true number of outliers in this selected set. In these experiments, ACODE is applied using a one-class support vector classifier to compute the conformity scores.
• Figure 4: Empirical 90th percentile of the 90% lower confidence bound on the number of outliers in the test set, computed by ACODE on synthetic data with adversarially hidden outliers exhibiting underdispersed conformity scores. Most local testing procedures cannot detect these outliers, but a data-driven approximation of the Shiraishi test enables our method to achieve high power. Other details are as in Figure \ref{['fig:exp-synthetic-1']}.
• Figure A1: Median computation time (milliseconds) for computing the overall lower bound $d$ across local tests, including a common preprocessing step (sorting pooled scores, computing ranks, and forming ordered conformal $p$-values). Times are medians over 10 repetitions.

Theorems & Definitions (17)

• Theorem 1
• Theorem 2
• Corollary 1
• Proposition 1: goemansolari2011
• Proposition A1
• proof
• Proposition A2
• proof
• Theorem A1: shiraishi1985local
• Proposition A3