A Comprehensive Forecasting-Based Framework for Time Series Anomaly Detection: Benchmarking on the Numenta Anomaly Benchmark (NAB)

TL;DR

This work tackles the lack of cross-domain evaluation in time series anomaly detection by proposing a unified forecasting-based framework that integrates classical methods (Holt-Winters, SARIMA) and deep-learning forecasters (LSTM, Informer) within a common residual-based detection interface. It conducts the first complete evaluation on the Numenta Anomaly Benchmark (NAB), covering 58 datasets across seven categories with 232 model training runs and 464 detection evaluations, achieving a 100% success rate. Key findings show that LSTM delivers the strongest overall performance on real-world data, Informer offers competitive accuracy with lower training time, and classical methods excel only on simple synthetic patterns; importantly, forecasting quality drives detection performance more than the choice of detection method. The results provide evidence-based guidance for practitioners (prefer LSTM for complex patterns, use Informer for efficiency, revert to classical methods for well-behaved seasonal data) and establish robust baselines to spur future forecasting-based anomaly detection research and reproducibility.

Abstract

Time series anomaly detection is critical for modern digital infrastructures, yet existing methods lack systematic cross-domain evaluation. We present a comprehensive forecasting-based framework unifying classical methods (Holt-Winters, SARIMA) with deep learning architectures (LSTM, Informer) under a common residual-based detection interface. Our modular pipeline integrates preprocessing (normalization, STL decomposition), four forecasting models, four detection methods, and dual evaluation through forecasting metrics (MAE, RMSE, PCC) and detection metrics (Precision, Recall, F1, AUC). We conduct the first complete evaluation on the Numenta Anomaly Benchmark (58 datasets, 7 categories) with 232 model training runs and 464 detection evaluations achieving 100\% success rate. LSTM achieves best performance (F1: 0.688, ranking first or second on 81\% of datasets) with exceptional correlation on complex patterns (PCC: 0.999). Informer provides competitive accuracy (F1: 0.683) with 30\% faster training. Classical methods achieve perfect predictions on simple synthetic data with 60 lower cost but show 2-3 worse F1-scores on real-world datasets. Forecasting quality dominates detection performance: differences between detection methods (F1: 0.621-0.688) are smaller than between forecasting models (F1: 0.344-0.688). Our findings provide evidence-based guidance: use LSTM for complex patterns, Informer for efficiency-critical deployments, and classical methods for simple periodic data with resource constraints. The complete implementation and results establish baselines for future forecasting-based anomaly detection research.

Figures (6)

• Figure 1: Proposed Methodology Pipeline for Time-Series Anomaly Detection. The workflow consists of five key stages: (1) Preprocessing (handling missing values, STL decomposition, normalization), (2) Forecasting Models (Holt-Winters, SARIMA, LSTM, Informer), (3) Residual Computation (forecasted vs. observed values), (4) Anomaly Detection (Z-test, Gaussian likelihood, IQR, Percentile thresholding), and (5) Evaluation Metrics (MAE, RMSE, PCC, DTW for forecasting; Precision, Recall, F1, AUC for detection). Arrows indicate the sequential flow and dependencies between stages.
• Figure 2: Model ranking heatmap across 58 NAB datasets sorted by difficulty (1=best, 4=worst for each dataset). LSTM (blue) achieves rank 1-2 on 81% of datasets, showing consistent reliability. Informer (orange) closely follows. Classical methods (SARIMA: green, Holt-Winters: red) rank poorly on complex datasets (top rows) but excel on simple synthetic patterns (bottom rows). Darker colors indicate better ranks.
• Figure 3: Detection metrics comparison across all model-method combinations. Forecasting model quality dominates detection performance, with LSTM and Informer consistently outperforming classical methods regardless of thresholding approach.
• Figure 4: F1-score heatmap across all 58 datasets and models. Color intensity indicates detection quality (green=high, red=low). LSTM and Informer maintain consistent high performance across diverse datasets, while classical methods struggle on complex real-world data.
• Figure 5: Comprehensive model comparison on NYC Taxi dataset. Top row: Forecasting performance showing ground truth (blue) versus model predictions (orange) with confidence intervals (shaded). Model names are clearly labeled above each subplot. Metrics (MAE, RMSE) are displayed in green boxes. Classical methods show significant errors: Holt-Winters (MAE: 156.397) produces a completely incorrect monotonic trend, while SARIMA (MAE: 1.006) captures some patterns but misses non-linear dynamics. Deep learning methods excel: LSTM (MAE: 0.123) and Informer (MAE: 0.111) accurately track the complex demand patterns. Bottom row: Anomaly detection using prediction errors as anomaly scores. Red shaded areas represent prediction errors, dashed lines indicate detection thresholds ($\mu + 2\sigma$), and red circles mark detected anomalies. Holt-Winters' poor forecasting leads to excessive false positives (anomalies everywhere). SARIMA detects 4 anomalies with moderate precision. LSTM identifies 4 anomalies with higher confidence, while Informer achieves the best balance with only 3 detected anomalies, demonstrating that superior forecasting accuracy directly translates to more precise anomaly detection with fewer false alarms.