Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World

TL;DR

The paper tackles the challenge of keeping anomaly detectors in AIOps up-to-date as operational data drifts over time. It systematically compares three retraining paradigms (static, full-history, sliding window) and two frequencies (blind versus drift-detected informed retraining), across two real-world datasets (Yahoo S5 and NAB) using five unsupervised detectors (FFT, SR, PCI, LSTM-AE, SR-CNN) and a FEDD drift detector. Key findings show that advanced models (LSTM-AE and SR-CNN) outperform simpler methods, sliding-window retraining benefits time-domain detectors like LSTM-AE while full-history can help domain-transforming detectors like SR-SCNN, and drift-detection–based retraining can improve performance over static baselines though periodic retraining often yields the best results. The work demonstrates that drift-aware maintenance pipelines are feasible and beneficial for real-world AIOps deployments, while highlighting the need for more open datasets and better drift detectors to generalize beyond the studied domains.

Abstract

Anomaly detection techniques are essential in automating the monitoring of IT systems and operations. These techniques imply that machine learning algorithms are trained on operational data corresponding to a specific period of time and that they are continuously evaluated on newly emerging data. Operational data is constantly changing over time, which affects the performance of deployed anomaly detection models. Therefore, continuous model maintenance is required to preserve the performance of anomaly detectors over time. In this work, we analyze two different anomaly detection model maintenance techniques in terms of the model update frequency, namely blind model retraining and informed model retraining. We further investigate the effects of updating the model by retraining it on all the available data (full-history approach) and only the newest data (sliding window approach). Moreover, we investigate whether a data change monitoring tool is capable of determining when the anomaly detection model needs to be updated through retraining.

Figures (6)

• Figure 1: The time series to the left of the vertical line represents the training data, while the one to the right represents the evaluation/testing data.
• Figure 2: Training and testing data in case of the static approach, full-history approach, and sliding window approach.
• Figure 3: Blind vs Informed Retraining.
• Figure 4: In this example we show the labels for 10 data points corresponding to one time series, where 1 indicates an anomaly and 0 indicates a non-anomalous point. The first row shows the ground truth, the second row shows the original predictions of one anomaly detection model and the third row shows the adjusted predictions considering a delay of 1. In the ground truth, there are 2 anomalous segments, each containing 3 anomalies. Since the model managed to predict the second anomaly in the sequence of anomalies and we tolerate a delay of 1 sample, the adjusted anomaly treats the entire first anomaly segment as a correct prediction. However, the second sequence of anomalies is treated as an incorrect prediction since even with a delay of one the anomaly on position 8 is not reported in time, while with a delay of 2, it would be reported in time. anomalyDetectionMicrosoft
• Figure 5: Delay metric applied to F1-score, precision and recall on Yahoo and NAB.