Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

System Log Parsing: A Survey

Tianzhu Zhang, Han Qiu, Gabriele Castellano, Myriana Rifai, Chung Shue Chen, Fabio Pianese

TL;DR

A comprehensive survey on log parsing is provided as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.

Abstract

Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analyzing them is laborious and error-prone. Consequently, many research endeavors have been devoted to automatic log analysis. However, these works typically expect structured input and struggle with the heterogeneous nature of raw system logs. Log parsing closes this gap by converting the unstructured system logs to structured records. Many parsers were proposed during the last decades to accommodate various log analysis applications. However, due to the ample solution space and lack of systematic evaluation, it is not easy for practitioners to find ready-made solutions that fit their needs. This paper aims to provide a comprehensive survey on log parsing. We begin with an exhaustive taxonomy of existing log parsers. Then we empirically analyze the critical performance and operational features for 17 open-source solutions both quantitatively and qualitatively, and whenever applicable discuss the merits of alternative approaches. We also elaborate on future challenges and discuss the relevant research directions. We envision this survey as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.

System Log Parsing: A Survey

TL;DR

A comprehensive survey on log parsing is provided as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.

Abstract

Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analyzing them is laborious and error-prone. Consequently, many research endeavors have been devoted to automatic log analysis. However, these works typically expect structured input and struggle with the heterogeneous nature of raw system logs. Log parsing closes this gap by converting the unstructured system logs to structured records. Many parsers were proposed during the last decades to accommodate various log analysis applications. However, due to the ample solution space and lack of systematic evaluation, it is not easy for practitioners to find ready-made solutions that fit their needs. This paper aims to provide a comprehensive survey on log parsing. We begin with an exhaustive taxonomy of existing log parsers. Then we empirically analyze the critical performance and operational features for 17 open-source solutions both quantitatively and qualitatively, and whenever applicable discuss the merits of alternative approaches. We also elaborate on future challenges and discuss the relevant research directions. We envision this survey as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.
Paper Structure (31 sections, 7 equations, 10 figures, 5 tables)

This paper contains 31 sections, 7 equations, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Log parsing in a nutshell
  3. What is a system log?
  4. Log parsing process
  5. Log parsing solutions: A taxonomy
  6. Clustering-based log parsing
  7. Hierarchical clustering
  8. Density-based clustering
  9. Online clustering
  10. Other clustering methods
  11. Frequent pattern mining
  12. Heuristic approach
  13. LCS-based approach
  14. Parsing tree
  15. Machine learning
  16. ...and 16 more sections

Figures (10)

  • Figure 1: A sample snippet of raw log entries along with the corresponding log printing statements in the source code.
  • Figure 2: The role of log parsing for system management.
  • Figure 3: Log parsers and the related works proposed during the past two decades.
  • Figure 4: A general overview of the log parsing process.
  • Figure 5: The average F-Measure for some log parsers extracted from the literature
  • ...and 5 more figures