VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection

TL;DR

The paper tackles the problem of evaluating time-series anomaly detectors with robust, threshold-independent metrics capable of handling range-based (subsequence) anomalies. It first confirms limitations of traditional point-based and threshold-based measures and demonstrates that AUC-based metrics, while threshold-independent, still falter under lag, noise, and varying anomaly cardinality. To address this, the authors introduce Range-AUC (range-aware ROC/PR) and the Volume Under the Surface (VUS) framework, which extends AUC to consider multiple buffer lengths and thresholds; they also propose two optimized VUS implementations (VUS_opt and VUS_opt_mem) to improve practicality. Extensive experiments across 10 datasets and 900 time-series show that VUS-based measures, particularly VUS-ROC, offer superior robustness, separability, and consistency compared with existing measures, albeit with higher offline computational cost than traditional AUC methods. The work provides a principled, scalable approach to evaluating time-series AD methods and suggests that VUS-based measures should be favored for reliable benchmarking and method comparison in practice.

Abstract

Anomaly detection (AD) is a fundamental task for time-series analytics with important implications for the downstream performance of many applications. In contrast to other domains where AD mainly focuses on point-based anomalies (i.e., outliers in standalone observations), AD for time series is also concerned with range-based anomalies (i.e., outliers spanning multiple observations). Nevertheless, it is common to use traditional point-based information retrieval measures, such as Precision, Recall, and F-score, to assess the quality of methods by thresholding the anomaly score to mark each point as an anomaly or not. However, mapping discrete labels into continuous data introduces unavoidable shortcomings, complicating the evaluation of range-based anomalies. Notably, the choice of evaluation measure may significantly bias the experimental outcome. Despite over six decades of attention, there has never been a large-scale systematic quantitative and qualitative analysis of time-series AD evaluation measures. This paper extensively evaluates quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios. Our results indicate that measures producing quality values independently of a threshold (i.e., AUC-ROC and AUC-PR) are more suitable for time-series AD. Motivated by this observation, we first extend the AUC-based measures to account for range-based anomalies. Then, we introduce a new family of parameter-free and threshold-independent measures, Volume Under the Surface (VUS), to evaluate methods while varying parameters. We also introduce two optimized implementations for VUS that reduce significantly the execution time of the initial implementation. Our findings demonstrate that our four measures are significantly more robust in assessing the quality of time-series AD methods.

Figures (19)

• Figure 1: Critical difference diagram computed with the Friedman test followed by a post-hoc Wilcoxon test (with $\alpha=0.1$) for the (a) F-score and (b) range-based F-score over $250$ time series in KDD21 kdd. Bold lines indicate insignificant differences of connected methods.
• Figure 2: Evaluation measures when we vary the (a) threshold, (b) lag, (c) noise, and (d) normal/abnormal ratio. Example with Isolation Forest methods over a snippet of an ECG time series goldberger_physiobank_2000.
• Figure 3: Influence of the anomaly detection method score (2) and labeling strategy (3), illustrated with three examples.
• Figure 4: Illustration of previous quality measures compared to our proposed measures. By varying the buffer window, VUS constructs a surface of TPR, FPR, and window. The volume under the surface is a measure of AUC for various windows.
• Figure 5: Illustration of proposed label extension strategy.