Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction

TL;DR

This work tackles log anomaly detection under limited labeling by introducing a content-and-causality aware feature extractor (DualGCN-LogAE) and an entirely unsupervised detector (Log2graphs). Logs are parsed and embedded with BERT, then organized into window-based graphs whose node/edge information is learned via a two-layer GCN autoencoder, optimized by a reconstruction-plus-regularization objective. Unlabeled anomaly detection is achieved through spectral clustering on compact log representations, with three clustering-quality metrics proposed to evaluate results without ground truth. Across five public log datasets, the approach yields superior log representations for classifiers and strong unsupervised detection performance, highlighting its practicality for scalable, label-free log analysis in diverse environments.

Abstract

In the era of rapid Internet development, log data has become indispensable for recording the operations of computer devices and software. These data provide valuable insights into system behavior and necessitate thorough analysis. Recent advances in text analysis have enabled deep learning to achieve significant breakthroughs in log anomaly detection. However, the high cost of manual annotation and the dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. It leverages the expressive power of large models for log content analysis and the capability of graph structures to encapsulate correlations between logs. It retains key log information while integrating the causal relationships between logs to achieve effective feature extraction. Additionally, we introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor. By employing graph clustering algorithms for log anomaly detection, Log2graphs enables the identification of abnormal logs without the need for labeled data. We comprehensively evaluate the feature extraction capability of DualGCN-LogAE and the anomaly detection performance of Log2graphs using public log datasets across five different scenarios. Our evaluation metrics include detection accuracy and graph clustering quality scores. Experimental results demonstrate that the log features extracted by DualGCN-LogAE outperform those obtained by other methods on classic classifiers. Moreover, Log2graphs surpasses existing unsupervised log detection methods, providing a robust tool for advancing log anomaly detection research.

Figures (11)

• Figure 1: Log2graphs is a comprehensive log anomaly detection framework consisting of three components: preprocessing of unstructured log data, feature extraction from structured logs, and unsupervised detection of anomalies.
• Figure 2: Log preprocessing consists of three key stages: parsing unstructured log data, vectorizing log content, and constructing causal graphs from the logs.
• Figure 3: It collaboratively compresses high-dimensional graph-structured log data into low-dimensional vector representations.
• Figure 4: The train loss of DualGCN-LogAE on five datasets.
• Figure 5: Flowchart of the spectral clustering algorithm. a. Construct the similarity matrix $S$ of the samples. b. Construct the adjacency matrix W and the degree matrix $D$ based on the similarity matrix $S$. c. Calculate and standardize the Laplacian matrix $L$. d. Spectral decomposition. e. Construct a new feature matrix. f. K-Means clustering.