Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Token Interdependency Parsing (Tipping) -- Fast and Accurate Log Parsing

Shayan Hashemi, Mika Mäntylä

TL;DR

Tipping presents a fast, accurate log parsing solution that unifies rule-based tokenization, interdependency graphs, and strongly connected components with map-reduce style parallelism to achieve near-linear scalability on multi-core CPUs. The method generates robust event templates and parameter masks through a two-phase process: information gathering and anchor-based clustering, followed by template and mask extraction. Across LogHub, LogPM, and LogLead benchmarks, Tipping demonstrates leading accuracy (GA, FGA, PA, PTA, RTA, FTA) and the fastest parsing speeds, with strong parallel performance on VM architectures. The downstream anomaly-detection studies show that parsing quality generally translates to improved detection performance in a dataset- and model-dependent manner. Overall, Tipping offers a scalable, reusable tool for large-scale automated log analysis with tangible benefits for anomaly detection and operational monitoring.

Abstract

In the last decade, an impressive increase in software adaptions has led to a surge in log data production, making manual log analysis impractical and establishing the necessity for automated methods. Conversely, most automated analysis tools include a component designed to separate log templates from their parameters, commonly referred to as a "log parser". This paper aims to introduce a new fast and accurate log parser, named "Tipping". Tipping combines rule-based tokenizers, interdependency token graphs, strongly connected components, and various techniques to ensure rapid, scalable, and precise log parsing. Furthermore, Tipping is parallelized and capable of running on multiple processing cores with close to linear efficiency. We evaluated Tipping against other state-of-the-art log parsers in terms of accuracy, performance, and the downstream task of anomaly detection. Accordingly, we found that Tipping outperformed existing methods in accuracy and performance in our evaluations. More in-depth, Tipping can parse 11 million lines of logs in less than 20 seconds on a laptop machine. Furthermore, we re-implemented a parallelized version of the past IpLom algorithm to demonstrate the effect of parallel processing, and it became the second-fastest parser. As logs keep growing in volume and complexity, the software engineering community needs to ensure automated log analysis tools keep up with the demand, being capable of efficiently handling massive volumes of logs with high accuracy. Tipping's robustness, versatility, efficiency, and scalability make it a viable tool for the modern automated log analysis task.

Token Interdependency Parsing (Tipping) -- Fast and Accurate Log Parsing

TL;DR

Tipping presents a fast, accurate log parsing solution that unifies rule-based tokenization, interdependency graphs, and strongly connected components with map-reduce style parallelism to achieve near-linear scalability on multi-core CPUs. The method generates robust event templates and parameter masks through a two-phase process: information gathering and anchor-based clustering, followed by template and mask extraction. Across LogHub, LogPM, and LogLead benchmarks, Tipping demonstrates leading accuracy (GA, FGA, PA, PTA, RTA, FTA) and the fastest parsing speeds, with strong parallel performance on VM architectures. The downstream anomaly-detection studies show that parsing quality generally translates to improved detection performance in a dataset- and model-dependent manner. Overall, Tipping offers a scalable, reusable tool for large-scale automated log analysis with tangible benefits for anomaly detection and operational monitoring.

Abstract

In the last decade, an impressive increase in software adaptions has led to a surge in log data production, making manual log analysis impractical and establishing the necessity for automated methods. Conversely, most automated analysis tools include a component designed to separate log templates from their parameters, commonly referred to as a "log parser". This paper aims to introduce a new fast and accurate log parser, named "Tipping". Tipping combines rule-based tokenizers, interdependency token graphs, strongly connected components, and various techniques to ensure rapid, scalable, and precise log parsing. Furthermore, Tipping is parallelized and capable of running on multiple processing cores with close to linear efficiency. We evaluated Tipping against other state-of-the-art log parsers in terms of accuracy, performance, and the downstream task of anomaly detection. Accordingly, we found that Tipping outperformed existing methods in accuracy and performance in our evaluations. More in-depth, Tipping can parse 11 million lines of logs in less than 20 seconds on a laptop machine. Furthermore, we re-implemented a parallelized version of the past IpLom algorithm to demonstrate the effect of parallel processing, and it became the second-fastest parser. As logs keep growing in volume and complexity, the software engineering community needs to ensure automated log analysis tools keep up with the demand, being capable of efficiently handling massive volumes of logs with high accuracy. Tipping's robustness, versatility, efficiency, and scalability make it a viable tool for the modern automated log analysis task.
Paper Structure (37 sections, 1 equation, 4 figures, 18 tables, 1 algorithm)

This paper contains 37 sections, 1 equation, 4 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Proposed Method
  4. Tokenizer
  5. Token Filter
  6. Occurences and Co-occurrences
  7. Interdependency Graph
  8. Strongly Dependent Token Group
  9. Anchor Tokens Selection
  10. Anchor-token-based Clustering
  11. Cluster Common Tokens
  12. Event Templates
  13. Parameter Masks
  14. Experiments
  15. LogHub Benchmark
  16. ...and 22 more sections

Figures (4)

  • Figure 1: An overall view of Tipping's components, internal workflow, relation, and execution order.
  • Figure 2: LogHub2k sensitivity experiment for $\theta$ values in the range of $[0.1, 1.0]$ and both with and without black and white special token regexes. The X-axis represents the $\theta$ value, while the Y-axis shows the respective metrics.
  • Figure 3: LogHub2.0 sensitivity experiment for $\theta$ values in the range of $[0.1, 1.0]$ and both with and without black and white special token regexes. The X-axis represents the $\theta$ value, while the Y-axis shows the respective metrics.
  • Figure 4: LogPM sensitivity experiment for $\theta$ values in the range of $[0.1, 1.0]$ and both with and without black and white special token regexes. The X-axis represents the $\theta$ value, while the Y-axis shows the respective metrics.