OneLog: Towards End-to-End Training in Software Log Anomaly Detection

TL;DR

OneLog introduces an end-to-end log anomaly detection approach that merges parsing, vectorization, and classification into a single Hierarchical CNN operating on raw character streams. Across six diverse datasets, OneLog achieves state-of-the-art $F_1$ scores, with near-perfect performance in most cases, and demonstrates that multi-project training can improve generalization when training data are limited. Cross-project experiments show that transfer is feasible primarily when source and target datasets are sufficiently similar. Interpretability analyses reveal that the model learns human-like parsing rules and multiple anomaly-detection strategies, suggesting practical viability and potential to reduce reliance on traditional log parsers.

Abstract

With the growth of online services, IoT devices, and DevOps-oriented software development, software log anomaly detection is becoming increasingly important. Prior works mainly follow a traditional four-staged architecture (Preprocessor, Parser, Vectorizer, and Classifier). This paper proposes OneLog, which utilizes a single Deep Neural Network (DNN) instead of multiple separate components. OneLog harnesses Convolutional Neural Networks (CNN) at the character level to take digits, numbers, and punctuations, which were removed in prior works, into account alongside the main natural language text. We evaluate our approach in six message- and sequence-based data sets: HDFS, Hadoop, BGL, Thunderbird, Spirit, and Liberty. We experiment with Onelog with single-, multi-, and cross-project setups. Onelog offers state-of-the-art performance in our datasets. Onelog can utilize multi-project datasets simultaneously during training, which suggests our model can generalize between datasets. Multi-project training also improves Onelog performance making it ideal when limited training data is available for an individual project. We also found that cross-project anomaly detection is possible with a single project pair (Liberty and Spirit). Analysis of model internals shows that one log has multiple modes of detecting anomalies and that the model learns manually validated parsing rules for the log messages. We conclude that character-based CNNs are a promising approach toward end-to-end learning in log anomaly detection. They offer good performance and generalization over multiple datasets. We will make our scripts publicly available upon the acceptance of this paper.

Figures (14)

• Figure 1: An overall view of the OneLog idea.
• Figure 2: The Hierarchical CNN model visualization. Variables $S$ and $E$ are indications of sequence and event lengths which are equivalents of $L_s$ and $L_l$ in the model explanation text. However, It must be noted that within our experiments, though the sequence events window size is 64, there is no limit to the number of characters in each log event (line).
• Figure 3: A sample from the Hadoop dataset where parameters are the sole determinants differentiating between normal and anomalous behaviors.
• Figure 4: A sample from the HDFS dataset where parameters are the sole determinants differentiating between normal and anomalous behaviors.
• Figure 5: The embedded vectors visualization of the HDFS log parsing benchmark dataset.