Prompting for Automatic Log Template Extraction

TL;DR

This paper tackles robust log parsing across diverse systems by removing the need for task-specific training. It introduces DivLog, a training-free framework that leverages in-context learning of large language models by constructing prompts from five most similar labeled candidates drawn from a diverse offline set via Determinantal Point Process sampling. The prompt guides the LLM to produce log templates where variables are replaced by <*> and constants preserved, enabling structure extraction without fine-tuning. Empirical evaluation on 16 LogPAI datasets shows state-of-the-art Parsing Accuracy of $98.1\%$, Template Precision of $92.1\%$, and Recall of $92.9\%$, with high stability across datasets, illustrating practical applicability in real-world log analysis.

Abstract

Log parsing, which involves log template extraction from semi-structured logs to produce structured logs, is the first and the most critical step in automated log analysis. However, current log parsers suffer from limited effectiveness for two reasons. First, traditional data-driven log parsers solely rely on heuristics or handcrafted features designed by domain experts, which may not consistently perform well on logs from diverse systems. Second, existing supervised log parsers require model tuning, which is often limited to fixed training samples and causes sub-optimal performance across the entire log source. To address this limitation, we propose DivLog, an effective log parsing framework based on the in-context learning (ICL) ability of large language models (LLMs). Specifically, before log parsing, DivLog samples a small amount of offline logs as candidates by maximizing their diversity. Then, during log parsing, DivLog selects five appropriate labeled candidates as examples for each target log and constructs them into a prompt. By mining the semantics of examples in the prompt, DivLog generates a target log template in a training-free manner. In addition, we design a straightforward yet effective prompt format to extract the output and enhance the quality of the generated log templates. We conducted experiments on 16 widely-used public datasets. The results show that DivLog achieves (1) 98.1% Parsing Accuracy, (2) 92.1% Precision Template Accuracy, and (3) 92.9% Recall Template Accuracy on average, exhibiting state-of-the-art performance.

Figures (10)

• Figure 1: A simple example of log parsing. The logging statements are typically not accessible in industrial scenarios.
• Figure 2: An example of ICL on commonsense reasoning task.
• Figure 3: The workflow of DivLog framework. The candidates need to be sampled and labeled before log parsing.
• Figure 4: DPP Sampler: Sampling the most diverse logs as the candidates of the prompt examples from offline logs for subsequent online parsing.
• Figure 5: KNN Selector: Selecting the most similar logs along with their labels as examples from the candidate set for the subsequent prompting.