Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speed and Performance of Parserless and Unsupervised Anomaly Detection Methods on Software Logs

Jesse Nyyssölä, Mika Mäntylä

TL;DR

This work tackles the problem of fast, unsupervised anomaly detection in software logs where labeled anomalies are scarce. It compares two simple custom models (Out-of-vocabulary Detector and Rarity Model) with two established methods (Isolation Forest and KMeans) across four public datasets and three log representations, under both unfiltered and normal-only training regimes, using $AUC-ROC$ as the primary performance metric and $F1$-scores as supplementary insight. The main findings show that Isolation Forest excels with unfiltered data, while OOVD with trigram representations performs best under normal-only training; RM and KMeans are generally slower or less accurate in several scenarios. The results offer practical guidance on method–representation–data regime tradeoffs for real-world log analysis and highlight the value of simple, fast baselines for scalable anomaly detection.

Abstract

Software log analysis can be laborious and time consuming. Time and labeled data are usually lacking in industrial settings. This paper studies unsupervised and time efficient methods for anomaly detection. We study two custom and two established models. The custom models are: an OOV (Out-Of-Vocabulary) detector, which counts the terms in the test data that are not present in the training data, and the Rarity Model (RM), which calculates a rarity score for terms based on their infrequency. The established models are KMeans and Isolation Forest. The models are evaluated on four public datasets (BGL, Thunderbird, Hadoop, HDFS) with three different representation techniques for the log messages (Words, character Trigrams, Parsed events). For training, we used both normal-only data, which is free of all anomalies, and unfiltered data, which contains both normal and anomalous instances. We used primarily the AUC-ROC metric for evaluation due to challenges in setting a threshold but we also include F1-scores for further insight. Different configurations are advised based on specific requirements. When training data is unfiltered, includes both normal and anomalous instances, the most effective combination is the Isolation Forest with event representation, achieving an AUC-ROC of 0.829. If it's possible to create a normal-only training dataset, combining the Out-Of-Vocabulary (OOV) detector with trigram representation yields the highest AUC-ROC of 0.846. For speed considerations, the OOV detector is optimal for filtered data, while the Rarity Model is the best choice for unfiltered data.

Speed and Performance of Parserless and Unsupervised Anomaly Detection Methods on Software Logs

TL;DR

This work tackles the problem of fast, unsupervised anomaly detection in software logs where labeled anomalies are scarce. It compares two simple custom models (Out-of-vocabulary Detector and Rarity Model) with two established methods (Isolation Forest and KMeans) across four public datasets and three log representations, under both unfiltered and normal-only training regimes, using as the primary performance metric and -scores as supplementary insight. The main findings show that Isolation Forest excels with unfiltered data, while OOVD with trigram representations performs best under normal-only training; RM and KMeans are generally slower or less accurate in several scenarios. The results offer practical guidance on method–representation–data regime tradeoffs for real-world log analysis and highlight the value of simple, fast baselines for scalable anomaly detection.

Abstract

Software log analysis can be laborious and time consuming. Time and labeled data are usually lacking in industrial settings. This paper studies unsupervised and time efficient methods for anomaly detection. We study two custom and two established models. The custom models are: an OOV (Out-Of-Vocabulary) detector, which counts the terms in the test data that are not present in the training data, and the Rarity Model (RM), which calculates a rarity score for terms based on their infrequency. The established models are KMeans and Isolation Forest. The models are evaluated on four public datasets (BGL, Thunderbird, Hadoop, HDFS) with three different representation techniques for the log messages (Words, character Trigrams, Parsed events). For training, we used both normal-only data, which is free of all anomalies, and unfiltered data, which contains both normal and anomalous instances. We used primarily the AUC-ROC metric for evaluation due to challenges in setting a threshold but we also include F1-scores for further insight. Different configurations are advised based on specific requirements. When training data is unfiltered, includes both normal and anomalous instances, the most effective combination is the Isolation Forest with event representation, achieving an AUC-ROC of 0.829. If it's possible to create a normal-only training dataset, combining the Out-Of-Vocabulary (OOV) detector with trigram representation yields the highest AUC-ROC of 0.846. For speed considerations, the OOV detector is optimal for filtered data, while the Rarity Model is the best choice for unfiltered data.
Paper Structure (17 sections, 4 figures, 6 tables)

This paper contains 17 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Research methods
  4. Preprocessing
  5. Datasets
  6. Metrics
  7. Models
  8. Out-of-vocabulary Detector
  9. Isolation Forest
  10. KMeans
  11. Rarity Model
  12. Results
  13. AUC-ROC results
  14. F1-score results
  15. Time results
  16. ...and 2 more sections

Figures (4)

  • Figure 1: The performance metrics on the Normal-only test setup on all datasets and representations
  • Figure 2: The performance metrics on the Unfiltered test setup on all datasets and representations
  • Figure 3: All the results plotted by AUC-ROC and F1-score
  • Figure 4: Distribution of the RM scores on BGL (Word-representation, filtered setup)