LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis

TL;DR

LogEval tackles the lack of standardized benchmarks for evaluating how large language models perform on log analysis tasks in AIOps. It introduces a four-task benchmark (log parsing, log anomaly detection, log fault diagnosis, log summarization) with 4,000 public logs per task and 15 bilingual prompts to enable fair, cross-model assessment of 18 LLMs under zero-shot and few-shot settings, including self-consistency analyses. The authors implement a data-quality pipeline (classification, standardization, bilingual prompts, diverse scenarios) and a multi-metric evaluation regime (accuracy, F1, ROUGE, cosine similarity, average tokens, inference time) to capture both correctness and efficiency. They demonstrate task- and language-dependent performance across models, highlighting GPT-4 and other API-based LLMs as strong performers in many settings, while also revealing limits in few-shot anomaly detection and model stability. LogEval thus provides actionable guidance for model selection and optimization in real-world log analysis, and outlines directions for future work, including task expansion, knowledge-graph integration, and privacy-preserving learning to advance LLMs in log analysis applications.

Abstract

Log analysis is crucial for ensuring the orderly and stable operation of information systems, particularly in the field of Artificial Intelligence for IT Operations (AIOps). Large Language Models (LLMs) have demonstrated significant potential in natural language processing tasks. In the AIOps domain, they excel in tasks such as anomaly detection, root cause analysis of faults, operations and maintenance script generation, and alert information summarization. However, the performance of current LLMs in log analysis tasks remains inadequately validated. To address this gap, we introduce LogEval, a comprehensive benchmark suite designed to evaluate the capabilities of LLMs in various log analysis tasks for the first time. This benchmark covers tasks such as log parsing, log anomaly detection, log fault diagnosis, and log summarization. LogEval evaluates each task using 4,000 publicly available log data entries and employs 15 different prompts for each task to ensure a thorough and fair assessment. By rigorously evaluating leading LLMs, we demonstrate the impact of various LLM technologies on log analysis performance, focusing on aspects such as self-consistency and few-shot contextual learning. We also discuss findings related to model quantification, Chinese-English question-answering evaluation, and prompt engineering. These findings provide insights into the strengths and weaknesses of LLMs in multilingual environments and the effectiveness of different prompt strategies. Various evaluation methods are employed for different tasks to accurately measure the performance of LLMs in log analysis, ensuring a comprehensive assessment. The insights gained from LogEvals evaluation reveal the strengths and limitations of LLMs in log analysis tasks, providing valuable guidance for researchers and practitioners.

Figures (16)

• Figure 1: The framework of LogEval
• Figure 2: Three examples of the processed questions
• Figure 3: The Accuracy in zero-shot Naive Q&A
• Figure 4: The Accuracy in few-shot Naive Q&A
• Figure 5: LogEval Overall Performance in zero-shot Naive Q&A