Unveiling the Flaws: A Critical Analysis of Initialization Effect on Time Series Anomaly Detection

TL;DR

Unveiling the Flaws investigates whether initialization effects undermine reported gains in time-series anomaly detection (TSAD). The authors conduct extensive experiments on SWaT and SMD with three representative TSAD methods—GDN, MTAD-GAT, and USAD—while systematically varying window size, seed, and normalization. They demonstrate high sensitivity and substantial variability driven by initialization, which can be exploited to artificially inflate performance. The work calls for rigorous evaluation protocols and transparent preprocessing reporting to ensure reliable, fair TSAD progress and practical applicability.

Abstract

Deep learning for time-series anomaly detection (TSAD) has gained significant attention over the past decade. Despite the reported improvements in several papers, the practical application of these models remains limited. Recent studies have cast doubt on these models, attributing their results to flawed evaluation techniques. However, the impact of initialization has largely been overlooked. This paper provides a critical analysis of the initialization effects on TSAD model performance. Our extensive experiments reveal that TSAD models are highly sensitive to hyperparameters such as window size, seed number, and normalization. This sensitivity often leads to significant variability in performance, which can be exploited to artificially inflate the reported efficacy of these models. We demonstrate that even minor changes in initialization parameters can result in performance variations that overshadow the claimed improvements from novel model architectures. Our findings highlight the need for rigorous evaluation protocols and transparent reporting of preprocessing steps to ensure the reliability and fairness of anomaly detection methods. This paper calls for a more cautious interpretation of TSAD advancements and encourages the development of more robust and transparent evaluation practices to advance the field and its practical applications.

Figures (4)

• Figure 1: F1 scores obtained by GDN on the SWAT dataset for varying window sizes.
• Figure 2: F1 scores obtained by GDN and MTAD-GAT on the SWAT dataset for varying random seed numbers.
• Figure 3: F1 scores obtained by USAD on the SWAT dataset for varying random seed numbers. The blue line is the USAD implementation which has normalization, whereas the orange line is USAD without using normalization.
• Figure 4: Density plots of anomaly scores obtained by GDN on the SMD dataset and MTAD on the SWAT dataset. The plot illustrates the distribution of anomaly scores across different sets: the training set, validation set, normal data from the test set, and abnormal data from the test set.