Formally Exploring Time-Series Anomaly Detection Evaluation Metrics

TL;DR

The paper addresses the problem that time-series anomaly detection (TSAD) evaluations suffer from inconsistent and potentially misleading metrics. It introduces a formal framework with two sets of verifiable properties (basic and advanced) to rigorously characterize TSAD evaluation metrics. Through analysis of 37 existing metrics, it shows that none satisfy all properties, explaining conflicting rankings in prior work. The authors then propose two general metrics, LARM and ALARM, that provably satisfy the advanced properties and offer adjustable trade-offs for application-specific needs. This work aims to enable trustworthy, interpretable, and robust evaluations of TSAD methods across diverse domains.

Abstract

Undetected anomalies in time series can trigger catastrophic failures in safety-critical systems, such as chemical plant explosions or power grid outages. Although many detection methods have been proposed, their performance remains unclear because current metrics capture only narrow aspects of the task and often yield misleading results. We address this issue by introducing verifiable properties that formalize essential requirements for evaluating time-series anomaly detection. These properties enable a theoretical framework that supports principled evaluations and reliable comparisons. Analyzing 37 widely used metrics, we show that most satisfy only a few properties, and none satisfy all, explaining persistent inconsistencies in prior results. To close this gap, we propose LARM, a flexible metric that provably satisfies all properties, and extend it to ALARM, an advanced variant meeting stricter requirements.

Figures (11)

• Figure 1: Illustration of a ground-truth anomaly (orange) and two prediction patterns (yellow). The upper prediction is timely and concise; the bottom prediction is delayed and oscillating, producing redundant alarms. Although both overlap equally with the true anomaly, they differ substantially in latency and redundancy—differences that many evaluation metrics ignore, often assigning them the same score.
• Figure 2: Challenging examples of prediction patterns (yellow) for different ground truths (orange) that common evaluation metrics struggle to order.
• Figure 3: The rankings of 16 synthetic methods assigned an individual color each from best (top) to worst (bottom) by the ALARM score (\ref{['theorem:advanced']}) (left) compared to rankings of 15 other metrics (right), reveal that no two metrics agree on a ranking.
• Figure 4: Visual motivation for the formalization of properties of TSAD evaluation metrics. Each panel shows a ground-truth anomaly (orange) and two prediction patterns (yellow), designed to isolate specific evaluation principles. With the exception of \ref{['prop:permutations']}, each property formalizes the notion that a metric would prefer the top prediction, when it detects anomalies more reliably (Props. \ref{['prop:detection']}, \ref{['prop:prediction_size']}), avoids redundancy (\ref{['prop:alarms']}), is more precise (\ref{['prop:false_positives']}), raises fewer false alarms (Props. \ref{['prop:false_positive_alarms']}, \ref{['prop:trust']}), or responds earlier (Props. \ref{['prop:temporal_order']}, \ref{['prop:early_bias']}).
• Figure 5: Rankings produced on a subset of SMD8 su2019robust of 20 methods wagner2023timesead by different metrics, illustrate their different priorities. Assuming the labels are correct, most methods produce a staggering number of false alarms almost approaching random predictions, which is severely penalized by both LARM and ALARM. Notably, both group predictions with similar behavior (e.g. rows 2,3, and 4 in the left figure) with similar scores, while classic metrics often fail to do so (see e.g. right figure).

Theorems & Definitions (87)

• Theorem 5.1
• Theorem 6.1
• Theorem 6.2
• Theorem A.1
• proof
• Lemma B.1
• proof
• Theorem B.2
• proof
• Theorem D.1