OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning

TL;DR

This work proposes OMLog, a semi-supervised online meta-learning method, to effectively tackle the distribution shift issue caused by changes in log event types and frequencies, and designs an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences in the feature space, thereby enhancing the generalization ability of the model to evolving data.

Abstract

Log anomaly detection (LAD) is essential to ensure safe and stable operation of software systems. Although current LAD methods exhibit significant potential in addressing challenges posed by unstable log events and temporal sequence patterns, their limitations in detection efficiency and generalization ability present a formidable challenge when dealing with evolving systems. To construct a real-time and reliable online log anomaly detection model, we propose OMLog, a semi-supervised online meta-learning method, to effectively tackle the distribution shift issue caused by changes in log event types and frequencies. Specifically, we introduce a maximum mean discrepancy-based distribution shift detection method to identify distribution changes in unseen log sequences. Depending on the identified distribution gap, the method can automatically trigger online fine-grained detection or offline fast inference. Furthermore, we design an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences in the feature space, thereby enhancing the generalization ability of the model to evolving data. Extensive experiments conducted on two publicly available log datasets, HDFS and BGL, validate the effectiveness of the OMLog approach. When trained using only normal log sequences, the proposed approach achieves the F1-Score of 93.7\% and 64.9\%, respectively, surpassing the performance of the state-of-the-art (SOTA) LAD methods and demonstrating superior detection efficiency.

Figures (6)

• Figure 1: An illustrative example of log terminology based on BGL dataset.
• Figure 2: Intuitive demonstration of distribution shifting from BGL log data. The datasets are constructed utilizing the sliding window with a window size of 100, a step size of 100, and is divided into 6 parts in chronological order. Initially, we randomly selected 200 samples from Part 1 to serve as the base part for comparison. Subsequently, we randomly selected 200 samples from each Part, where each subset comprised 100 normal samples and 100 abnormal samples. Subfigures (a)-(f) show examples of normal samples and (g)-(l) show examples of abnormal samples.
• Figure 3: Distribution shift between batches of data in the BGL dataset.
• Figure 4: Sample similarity of the BGL dataset based on dynamic Time Warping. Internal similarity, i.e., the average of the similarity between each sample within a batch of data; External similarity, i.e., the similarity between each sample of the current batch of data and each sample of the previous 10 batches of data.
• Figure 5: Overview of OMLog approach. OMLog implements semi-supervised learning based on next-event prediction, with an auto-encoder for the normality detection model and an LSTM for the anomaly detection model. The novel components of OMLog are DSD and OMD.