LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, Shaowei Wang
TL;DR
This paper investigates using generative Large Language Models (LLMs) for log parsing by introducing LLMParser, a framework that leverages few-shot tuning to translate raw logs into templates. Across 16 open-source systems, LLMParser achieves near-perfect Parsing Accuracy ($$PA$$) around $0.95$-$0.96$, outperforming traditional parsers like Drain and Logram and rivaling LogPPT in grouping quality. The authors compare multiple open-source LLMs (text2text and text generation) and show that smaller models can match larger ones in accuracy while offering faster inference; in-context learning underperforms compared to few-shot tuning. They also analyze training size, model size, and cross-system pre-training, revealing that diverse, task-relevant sampling and tuning data are key, while external pre-training often hurts performance. The findings provide empirical support for LLM-based log parsing and point to future work on sampling strategies, efficiency, and generalization to unseen log templates.
Abstract
Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.