Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correction

Yifan Wu, Siyu Yu, Ying Li

TL;DR

AdaParser addresses the dual challenges of accuracy and adaptability in log parsing by combining a tree-based template store with a self-generated in-context learning (SG-ICL) strategy and a novel template correction mechanism. The framework maintains a dynamic candidate set of LLM-generated templates and uses a template corrector to fix plausible and broad template errors, enabling robust parsing as logs evolve. Empirical results on Loghub-2.0 show state-of-the-art performance across grouping and template metrics, with strong zero-shot performance and improved resilience across different LLMs. The approach is efficient, privacy-friendly (with open-source LLMs possible via local serving), and practical for real-world deployment in evolving software systems.

Abstract

Log parsing transforms log messages into structured formats, serving as a crucial step for log analysis. Despite a variety of log parsers that have been proposed, their performance on evolving log data remains unsatisfactory due to reliance on human-crafted rules or learning-based models with limited training data. The recent emergence of large language models (LLMs) has demonstrated strong abilities in understanding natural language and code, making it promising to apply LLMs for log parsing. Consequently, several studies have proposed LLM-based log parsers. However, LLMs may produce inaccurate templates, and existing LLM-based log parsers directly use the template generated by the LLM as the parsing result, hindering the accuracy of log parsing. Furthermore, these log parsers depend heavily on historical log data as demonstrations, which poses challenges in maintaining accuracy when dealing with scarce historical log data or evolving log data. To address these challenges, we propose AdaParser, an effective and adaptive log parsing framework using LLMs with self-generated in-context learning (SG-ICL) and self-correction. To facilitate accurate log parsing, AdaParser incorporates a novel component, a template corrector, which utilizes the LLM to correct potential parsing errors in the templates it generates. In addition, AdaParser maintains a dynamic candidate set composed of previously generated templates as demonstrations to adapt evolving log data. Extensive experiments on public large-scale datasets indicate that AdaParser outperforms state-of-the-art methods across all metrics, even in zero-shot scenarios. Moreover, when integrated with different LLMs, AdaParser consistently enhances the performance of the utilized LLMs by a large margin.

Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correction

TL;DR

AdaParser addresses the dual challenges of accuracy and adaptability in log parsing by combining a tree-based template store with a self-generated in-context learning (SG-ICL) strategy and a novel template correction mechanism. The framework maintains a dynamic candidate set of LLM-generated templates and uses a template corrector to fix plausible and broad template errors, enabling robust parsing as logs evolve. Empirical results on Loghub-2.0 show state-of-the-art performance across grouping and template metrics, with strong zero-shot performance and improved resilience across different LLMs. The approach is efficient, privacy-friendly (with open-source LLMs possible via local serving), and practical for real-world deployment in evolving software systems.

Abstract

Log parsing transforms log messages into structured formats, serving as a crucial step for log analysis. Despite a variety of log parsers that have been proposed, their performance on evolving log data remains unsatisfactory due to reliance on human-crafted rules or learning-based models with limited training data. The recent emergence of large language models (LLMs) has demonstrated strong abilities in understanding natural language and code, making it promising to apply LLMs for log parsing. Consequently, several studies have proposed LLM-based log parsers. However, LLMs may produce inaccurate templates, and existing LLM-based log parsers directly use the template generated by the LLM as the parsing result, hindering the accuracy of log parsing. Furthermore, these log parsers depend heavily on historical log data as demonstrations, which poses challenges in maintaining accuracy when dealing with scarce historical log data or evolving log data. To address these challenges, we propose AdaParser, an effective and adaptive log parsing framework using LLMs with self-generated in-context learning (SG-ICL) and self-correction. To facilitate accurate log parsing, AdaParser incorporates a novel component, a template corrector, which utilizes the LLM to correct potential parsing errors in the templates it generates. In addition, AdaParser maintains a dynamic candidate set composed of previously generated templates as demonstrations to adapt evolving log data. Extensive experiments on public large-scale datasets indicate that AdaParser outperforms state-of-the-art methods across all metrics, even in zero-shot scenarios. Moreover, when integrated with different LLMs, AdaParser consistently enhances the performance of the utilized LLMs by a large margin.
Paper Structure (37 sections, 1 equation, 7 figures, 3 tables)

This paper contains 37 sections, 1 equation, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. background and motivation
  3. Log Parsing
  4. Large Language Models
  5. Challenges of Existing LLM-based parsers
  6. Unsatisfactory accuracy in parsing logs
  7. Poor adaptability to evolving logs
  8. Approach
  9. Overview
  10. Tree-based Log Parser
  11. SG-ICL-based Parser
  12. Template Corrector
  13. Template Matching Correction
  14. Variable Abstracting Correction
  15. Experimental Setup
  16. ...and 22 more sections

Figures (7)

  • Figure 1: An example of log parsing from Spark.
  • Figure 2: Two types of parsing errors in LLM-based parser.
  • Figure 3: The overview of AdaParser framework.
  • Figure 4: The workflow of SG-ICL-based parser. After step $t$, the message with the template generated by the LLM is added to the candidate set and then used as a demonstration at step $t+1$.
  • Figure 5: Two examples of self-correction. Template matching correction and variable abstracting correction are used to correct the two types of parsing errors. The corrected parts are marked in red.
  • ...and 2 more figures