RESTAD: REconstruction and Similarity based Transformer for time series Anomaly Detection

TL;DR

RESTAD addresses unsupervised time-series anomaly detection by augmenting a Transformer with a learnable RBF layer that computes similarity to normal latent patterns. The RBF outputs are fused multiplicatively with reconstruction error to form a robust composite anomaly score, improving sensitivity to subtle anomalies. Across multiple public benchmarks, RESTAD outperforms state-of-the-art baselines with robustness to initialization and RBF placement, highlighting the value of nonparametric density alignment in latent space. This approach has practical impact for detecting subtle deviations in complex time-series without labeled anomalies, and suggests broader applicability of RBF augmentation in deep sequence models.

Abstract

Anomaly detection in time series data is crucial across various domains. The scarcity of labeled data for such tasks has increased the attention towards unsupervised learning methods. These approaches, often relying solely on reconstruction error, typically fail to detect subtle anomalies in complex datasets. To address this, we introduce RESTAD, an adaptation of the Transformer model by incorporating a layer of Radial Basis Function (RBF) neurons within its architecture. This layer fits a non-parametric density in the latent representation, such that a high RBF output indicates similarity with predominantly normal training data. RESTAD integrates the RBF similarity scores with the reconstruction errors to increase sensitivity to anomalies. Our empirical evaluations demonstrate that RESTAD outperforms various established baselines across multiple benchmark datasets.

Figures (6)

• Figure 1: Comparison of traditional reconstruction and RBF-enhanced anomaly detection: a) Original signal with subtle and significant anomalies compared to its reconstruction. b) Reconstruction errors for the signals in (a), highlighting challenges in detecting subtle anomalies. c) Visualization of a model integrated with an RBF, shown via a 2D scatter plot that includes typical data, subtle and significant anomalies, and the RBF center with its influence radius, showing the RBF's ability to differentiate typical points from anomalies. d) Enhanced anomaly score using the RBF, which shows improved detection of subtle anomalies.
• Figure 2: Overview of the proposed RESTAD model. Here, the RBF layer is added after the second encoder layer.
• Figure 3: Anomaly scores of different models for a segment of SMD dataset. The highlighted regions in red indicate the true anomaly periods (labeled by an expert).
• Figure 4: Effect of our composite anomaly score ($\epsilon_r$$\times$$\epsilon_s$) compared to reconstruction error ($\epsilon_r$) across segments of all datasets. The highlighted regions in red indicate the true anomaly periods (labeled by an expert).
• Figure 5: RESTAD Performance with varying RBF layer placements.