Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, Michael R. Lyu

TL;DR

A thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting is conducted, and a new evaluation metric is introduced to mitigate the sensitivity of existing metrics to imbalanced data distributions.

Abstract

Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.

A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

TL;DR

A thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting is conducted, and a new evaluation metric is introduced to mitigate the sensitivity of existing metrics to imbalanced data distributions.

Abstract

Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.
Paper Structure (31 sections, 5 figures, 1 table)

This paper contains 31 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Existing Log Parsing Techniques
  4. Motivation
  5. Dataset Construction
  6. Overview
  7. Preprocessing
  8. Log Grouping
  9. Template Annotation
  10. Log-Template Matching
  11. Template Refinement
  12. Annotation Results
  13. Study Design
  14. Research Question
  15. Evaluation Metrics
  16. ...and 16 more sections

Figures (5)

  • Figure 1: The overall framework of data annotation
  • Figure 2: Distribution comparison of template frequencies and parameter counts in Loghub-2.0 and Loghub-2k
  • Figure 3: The evaluation results of all log parsers on Loghub-2k and Loghub-2.0
  • Figure 4: The evaluation results of log parsers on logs with different frequencies
  • Figure 5: The evaluation results of log parsers on logs with different parameter counts