Active Hypothesis Testing for Correlated Combinatorial Anomaly Detection

TL;DR

This work tackles the problem of identifying a small anomalous subset of streams under correlated noise with a limited measurement budget. It introduces ECC-AHT, a correlation-aware, continuous-action paradigm that designs measurements to maximize Chernoff information between competing hypotheses, effectively canceling shared noise through differential sensing. The authors establish fundamental information-theoretic limits, finite-sample guarantees, and asymptotic optimality, showing that correlation, when leveraged properly, increases the attainable information rate $\Gamma^*$ and yields near-optimal sample complexity. Empirically, ECC-AHT outperforms baselines in synthetic settings and real-world WaDi data, with ablations highlighting the necessity of both correlation-aware design and pairwise champion–challenger testing. The work provides a principled, scalable approach to high-dimensional anomaly detection with practical impact for cyber-physical systems and large-scale monitoring.

Abstract

We study the problem of identifying an anomalous subset of streams under correlated noise, motivated by monitoring and security in cyber-physical systems. This problem can be viewed as a form of combinatorial pure exploration, where each stream plays the role of an arm and measurements must be allocated sequentially under uncertainty. Existing combinatorial bandit and hypothesis testing methods typically assume independent observations and fail to exploit correlation for efficient measurement design. We propose ECC-AHT, an adaptive algorithm that selects continuous, constrained measurements to maximize Chernoff information between competing hypotheses, enabling active noise cancellation through differential sensing. ECC-AHT achieves optimal sample complexity guarantees and significantly outperforms state-of-the-art baselines in both synthetic and real-world correlated environments. The code is available on https://github.com/VincentdeCristo/ECC-AHT

Figures (26)

• Figure 1: Visualization of ECC-AHT dynamics on a toy instance ($K=15$, $n=3$, $\rho=0.6$). (a)--(d) Evolution of marginal beliefs $p_t(k)$ from uniform priors to the successful isolation of anomalies. (e) Heatmap of action weights $c_t[k]$. (f) Snapshot of the action vector $c_t$ at $t=5$ overlaying the correlation structure $\Sigma[i^\star, :]$, illustrating how the algorithm leverages correlation for variance reduction. See Appendix \ref{['app:interpretation']} for a detailed interpretative analysis.
• Figure 2: Ablation analysis.$K=100, n=3, \rho=0.8, B=4.0$. The performance gap between ECC-AHT and No-QP confirms the importance of optimization-based experimental design.
• Figure 3: Comparison with state-of-the-art. ECC-AHT outperforms HDS, CombGapE and matches randomized TTTS while remaining deterministic.
• Figure 4: Real-world results on WaDi. Exploiting correlation reduces detection delay and improves end-to-end performance.
• Figure 5: Scalability and correlation exploitation. F1-score versus samples. ECC-AHT scales to high dimensions and benefits increasingly from stronger correlation. Shaded regions show $95\%$ confidence intervals.

Theorems & Definitions (40)

• Remark 4.1
• Theorem 4.2: Order-Optimal Non-Asymptotic Sample Complexity
• Theorem 4.3: Exact Asymptotic Optimality
• Remark 5.1
• Remark 5.2
• Remark 5.3
• Theorem 3.1: Empirical Sufficient Condition via Effective Rank
• proof : Proof Sketch
• Remark 3.2: Impossibility under vanishing effective rank
• Corollary 3.3: Regularization restores effective rank