Self-Supervised Anomaly Detection in the Wild: Favor Joint Embeddings Methods

TL;DR

A comprehensive evaluation of SSL methods for real-world anomaly detection, focusing on sewer infrastructure, and highlights the superiority of joint-embedding methods like SimCLR and Barlow Twins over reconstruction-based approaches such as MAE, which struggle to maintain performance under class imbalance.

Abstract

Accurate anomaly detection is critical in vision-based infrastructure inspection, where it helps prevent costly failures and enhances safety. Self-Supervised Learning (SSL) offers a promising approach by learning robust representations from unlabeled data. However, its application in anomaly detection remains underexplored. This paper addresses this gap by providing a comprehensive evaluation of SSL methods for real-world anomaly detection, focusing on sewer infrastructure. Using the Sewer-ML dataset, we evaluate lightweight models such as ViT-Tiny and ResNet-18 across SSL frameworks, including BYOL, Barlow Twins, SimCLR, DINO, and MAE, under varying class imbalance levels. Through 250 experiments, we rigorously assess the performance of these SSL methods to ensure a robust and comprehensive evaluation. Our findings highlight the superiority of joint-embedding methods like SimCLR and Barlow Twins over reconstruction-based approaches such as MAE, which struggle to maintain performance under class imbalance. Furthermore, we find that the SSL model choice is more critical than the backbone architecture. Additionally, we emphasize the need for better label-free assessments of SSL representations, as current methods like RankMe fail to adequately evaluate representation quality, making cross-validation without labels infeasible. Despite the remaining performance gap between SSL and supervised models, these findings highlight the potential of SSL to enhance anomaly detection, paving the way for further research in this underexplored area of SSL applications.

Figures (10)

• Figure 1: Samples from the Sewer-ML dataset harum2021sewer. The red circle highlights a defect associated with lateral reinstatement cuts, where improper cutting or misalignment can cause issues such as blockages, leaks, or structural weakness in the sewer system (for further details on defect types and image samples used for training and validation, refer to Appendix \ref{['app:train-val-image-samples']}).
• Figure 2: ResNet-18 validation performance heatmaps across imbalance levels. The x-axis represents the imbalance levels in the validation set, while the y-axis indicates the method and the imbalance level used during training.
• Figure 3: Val F1 Score by min_scale and t_val. This heatmap shows the performance variation in terms of F1 score, demonstrating the interaction between these two hyperparameters.
• Figure 4: ResNet-18's F1, F2$_{CIW}$, and F1$_{Normal}$ validation score heatmaps across imbalance levels. The x-axis represents the imbalance levels in the validation set, while the y-axis indicates the method and the imbalance level used during training.
• Figure 5: ResNet-18's precision, recall, and f1 validation score heatmaps across imbalance levels. The x-axis represents the imbalance levels in the validation set, while the y-axis indicates the method and the imbalance level used during training.