LogSD: Detecting Anomalies from System Logs through Self-supervised Learning and Frequency-based Masking

TL;DR

LogSD tackles semi-supervised log anomaly detection by addressing the bias toward frequent log events. It introduces a dual-network self-supervised framework with frequency-based masking and a global-to-local reconstruction paradigm, optimizing three losses ($L_r$, $L_o$, $L_p$) to derive an anomaly score from representation discrepancies, $L_p$ being the primary indicator. Across HDFS, BGL, and Spirit, LogSD consistently outperforms eight state-of-the-art baselines, with notable gains in F1-score and reduced variance due to the frequency-focused training. The approach demonstrates that concentrating on infrequent log messages yields less biased normal-pattern representations and improves discriminative capability for anomaly detection in large-scale log systems.

Abstract

Log analysis is one of the main techniques that engineers use for troubleshooting large-scale software systems. Over the years, many supervised, semi-supervised, and unsupervised log analysis methods have been proposed to detect system anomalies by analyzing system logs. Among these, semi-supervised methods have garnered increasing attention as they strike a balance between relaxed labeled data requirements and optimal detection performance, contrasting with their supervised and unsupervised counterparts. However, existing semi-supervised methods overlook the potential bias introduced by highly frequent log messages on the learned normal patterns, which leads to their less than satisfactory performance. In this study, we propose LogSD, a novel semi-supervised self-supervised learning approach. LogSD employs a dual-network architecture and incorporates a frequency-based masking scheme, a global-to-local reconstruction paradigm and three self-supervised learning tasks. These features enable LogSD to focus more on relatively infrequent log messages, thereby effectively learning less biased and more discriminative patterns from historical normal data. This emphasis ultimately leads to improved anomaly detection performance. Extensive experiments have been conducted on three commonly-used datasets and the results show that LogSD significantly outperforms eight state-of-the-art benchmark methods.

Figures (5)

• Figure 1: Illustrative example of the issue identified and the solution provided by our proposed method, LogSD. In both sub-figures, $S_1$ to $S5$ are normal training sequences. For testing, $S_6$ represents a normal sequence, and $S_a$ is anomalous. Sub-figure (a) demonstrates how frequent event $E_5$ in training sequences $S_2$ to $S_5$ can obscure the distinction between normal sequence and anomalous sequence in latent space; Sub-figure (b) depicts how LogSD addresses this issue by utilizing the contrast between two subnets: one to emphasize infrequent events and the other to focus on frequent events.
• Figure 2: A snippet of HDFS (Hadoop Distributed File System) raw logs and events after parsing
• Figure 3: Different Reconstruction Paradigms.
• Figure 4: Sensitivity analysis on BGL under 100 logs window setting.
• Figure 5: Scalability analysis on BGL under 100 logs window setting.