A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection

TL;DR

This work addresses the misalignment between TSAD evaluation and operational objectives in IoT settings. It proposes a problem‑oriented taxonomy that reinterprets over twenty metrics across six dimensions, focusing on comparability, robustness, timeliness, and cost awareness rather than purely mathematical form. Through synthetic experiments with genuine, random, and oracle detectors, the authors quantify how well each metric discriminates meaningful detections from noise and reveal that some popular metrics (e.g., NAB, Point‑Adjust) can inflate scores under random scoring. The framework provides practical guidance for selecting or designing context‑aware, robust, and fair evaluation methods and highlights the need for multi‑objective standards that balance timely detection, human‑audit costs, and annotation uncertainty. Overall, the work offers a unified perspective that clarifies metric diversity and informs principled metric design for TSAD in heterogeneous IoT environments.

Abstract

Time series anomaly detection is widely used in IoT and cyber-physical systems, yet its evaluation remains challenging due to diverse application objectives and heterogeneous metric assumptions. This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address, rather than their mathematical forms or output structures. We categorize over twenty commonly used metrics into six dimensions: 1) basic accuracy-driven evaluation; 2) timeliness-aware reward mechanisms; 3) tolerance to labeling imprecision; 4) penalties reflecting human-audit cost; 5) robustness against random or inflated scores; and 6) parameter-free comparability for cross-dataset benchmarking. Comprehensive experiments are conducted to examine metric behavior under genuine, random, and oracle detection scenarios. By comparing their resulting score distributions, we quantify each metric's discriminative ability -- its capability to distinguish meaningful detections from random noise. The results show that while most event-level metrics exhibit strong separability, several widely used metrics (e.g., NAB, Point-Adjust) demonstrate limited resistance to random-score inflation. These findings reveal that metric suitability must be inherently task-dependent and aligned with the operational objectives of IoT applications. The proposed framework offers a unified analytical perspective for understanding existing metrics and provides practical guidance for selecting or developing more context-aware, robust, and fair evaluation methodologies for time series anomaly detection.

Figures (5)

• Figure 1: Problem-oriented Taxonomy of Time Series Anomaly Detection Metrics. This figure illustrates the proposed problem-oriented taxonomy of anomaly detection metrics. The framework is organized along two principal dimensions: a Foundational dimension emphasizing comparability and robustness (e.g., parameter-free and label-robust metrics), and an Application-oriented dimension reflecting task-specific evaluation goals (e.g., early-reward, cost-awareness, and robustness to random scoring). Each subcategory is associated with representative metrics, color-coded by their granularity — point-based (yellow), event-based (green), or hybrid (blue). This hierarchical structure highlights how different metrics are designed to address distinct evaluation challenges and reveals their conceptual interrelations.
• Figure 2: Concentrated vs. Distributed False Alarms under Batch-level Evaluation. Illustration of how $\mathit{LSF}^w$ differentiates between concentrated and distributed false alarms. Although both scenarios contain three point-wise false positives, the batch-level penalty differs: only one window is affected in Scenario 1, whereas three windows are affected in Scenario 2. This leads to a higher penalty for distributed false alarms.
• Figure 3: Distribution of metric scores under genuine detectors, random guessing, and oracle-based attacks. Metrics whose genuine and random score distributions largely overlap are considered less robust and should be used cautiously or in combination with other stable metrics. Note that since the score ranges of TD and NAB differ from typical F1-based metrics (not bounded within [0, 1]), the NAB scores are normalized to [0, 1], and the TD scores are inversely normalized because smaller distances indicate better performance.
• Figure 4: Comprehensive comparison of 20+ time series anomaly detection metrics based on average effect size, AUC, genuine/random scores, and monotonicity. Higher intensity indicates stronger separability between genuine and random detection results. Here, Rn% denote datasets with overall anomaly ratios of n%, respectively, where each anomaly set contains diverse anomaly types.
• Figure 5: Joint analysis of effect size and AUC across all metrics. Metrics located in the left region exhibit high discriminative ability, clearly separating genuine and random scores, while those near the right indicate weak robustness to random guessing.