Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction

TL;DR

This study systematically evaluates deep learning approaches for log-based failure prediction by pairing log sequence embeddings with four DL encoders (LSTM, BiLSTM, CNN, Transformer) and three embedding strategies (Logkey2vec, BERT, FastText+TF-IDF) within a modular architecture. It introduces a controlled synthetic data-generation pipeline driven by deterministic automata models and regular-expression failure patterns to vary dataset size, log sequence length, failure rate, and pattern type, producing 360 datasets per behaviour model; results indicate CNN-based encoders with Logkey2vec embeddings achieve the best overall accuracy (F1) and robustness, especially for larger datasets or higher failure percentages. The study also compares against a strong traditional ML baseline (Random Forest), finding DL configurations substantially outperforming it, and validates findings on a real OpenStack dataset (OpenStack_PF) with results closely matching synthetic data. Practical guidelines are derived: for dataset size above 3000 or failure rate above 7.5%, CNN+L is recommended; when smaller datasets with low failure rates are used, BiLSTM+B may be preferable; and BERT generally provides strong semantic embeddings across encoders. The replication package supports further benchmarking and extension, reinforcing the practical relevance for AIOps and software engineering research.

Abstract

With the increasing complexity and scope of software systems, their dependability is crucial. The analysis of log data recorded during system execution can enable engineers to automatically predict failures at run time. Several Machine Learning (ML) techniques, including traditional ML and Deep Learning (DL), have been proposed to automate such tasks. However, current empirical studies are limited in terms of covering all main DL types -- Recurrent Neural Network (RNN), Convolutional Neural network (CNN), and transformer -- as well as examining them on a wide range of diverse datasets. In this paper, we aim to address these issues by systematically investigating the combination of log data embedding strategies and DL types for failure prediction. To that end, we propose a modular architecture to accommodate various configurations of embedding strategies and DL-based encoders. To further investigate how dataset characteristics such as dataset size and failure percentage affect model accuracy, we synthesised 360 datasets, with varying characteristics, for three distinct system behavioral models, based on a systematic and automated generation approach. Using the F1 score metric, our results show that the best overall performing configuration is a CNN-based encoder with Logkey2vec. Additionally, we provide specific dataset conditions, namely a dataset size >350 or a failure percentage >7.5%, under which this configuration demonstrates high accuracy for failure prediction.

Figures (14)

• Figure 1: An example illustrating the concepts of log, log message, log template, and log sequence
• Figure 2: Illustration of Log Analysis Tasks.
• Figure 3: Comparison of Normal Sequence (on the left) and Positive Sequences in Log Analysis Tasks (on the right).
• Figure 4: Overview of the modular architecture for failure prediction
• Figure 5: An example of a behaviour model.