Understanding the limitations of self-supervised learning for tabular anomaly detection

TL;DR

The paper evaluates self-supervised learning for tabular anomaly detection across 26 ODDS datasets and finds that SSL representations rarely outperform raw features for shallow detectors in a one-class setting. A key finding is that neural networks tend to introduce irrelevant directions, degrading anomaly detectors, though restricting attention to a residual subspace of the neural representations can recover performance. The study comprehensively benchmarks various pretext tasks, architectures, and losses, revealing nuanced effects: some SSL tasks help with local or dependency-structure anomalies, but global anomalies are generally poorly captured. Practically, the work suggests prioritising the original feature space and, when using SSL, leveraging residual subspaces and task-specific configurations; k-NN remains a strong baseline in many cases. These results inform when SSL is advantageous in tabular domains and motivate further research into regularisation and hybrid approaches for tabular anomaly detection.

Abstract

While self-supervised learning has improved anomaly detection in computer vision and natural language processing, it is unclear whether tabular data can benefit from it. This paper explores the limitations of self-supervision for tabular anomaly detection. We conduct several experiments spanning various pretext tasks on 26 benchmark datasets to understand why this is the case. Our results confirm representations derived from self-supervision do not improve tabular anomaly detection performance compared to using the raw representations of the data. We show this is due to neural networks introducing irrelevant features, which reduces the effectiveness of anomaly detectors. However, we demonstrate that using a subspace of the neural network's representation can recover performance.

Figures (19)

• Figure 1: Self-supervised anomaly detection workflow. The data are only augmented and fed through the projector during training.
• Figure 2: Box plot comparing nearest neighbour AUROCs for each of the embeddings, ordered by median performance. For each self-supervised task, we filter the results by architecture and loss function to include the embedding with the best-performing results.
• Figure 3: Critical difference diagram comparing the embeddings in a pairwise manner. The horizontal scale denotes the average rank of each embedding. The dark lines between different detectors indicate a statistical difference ($p < 0.05$) in results when running pairwise comparison tests. The baseline scores greatly outrank the pretext tasks. In contrast, the scores among the pretext tasks are more closely aligned.
• Figure 4: Box plot comparing detector performance on the self-supervised embeddings.
• Figure 5: Bar chart comparing baseline and self-supervised embedding results on HTTP.