Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Studying and Benchmarking Large Language Models For Log Level Suggestion

Yi Wen Heng, Zeyang Ma, Zhenhao Li, Dong Jae Kim, Tse-Hsun, Chen

TL;DR

A detailed empirical analysis is undertaken to investigate the impact of characteristics and learning paradigms on the performance of 12 open-source LLMs in log level suggestion, indicating that although smaller LLMs can perform effectively with appropriate instruction and suitable techniques, there is still considerable potential for improvement in their ability to suggest log levels.

Abstract

Large Language Models (LLMs) have become a focal point of research across various domains, including software engineering, where their capabilities are increasingly leveraged. Recent studies have explored the integration of LLMs into software development tools and frameworks, revealing their potential to enhance performance in text and code-related tasks. Log level is a key part of a logging statement that allows software developers control the information recorded during system runtime. Given that log messages often mix natural language with code-like variables, LLMs' language translation abilities could be applied to determine the suitable verbosity level for logging statements. In this paper, we undertake a detailed empirical analysis to investigate the impact of characteristics and learning paradigms on the performance of 12 open-source LLMs in log level suggestion. We opted for open-source models because they enable us to utilize in-house code while effectively protecting sensitive information and maintaining data security. We examine several prompting strategies, including Zero-shot, Few-shot, and fine-tuning techniques, across different LLMs to identify the most effective combinations for accurate log level suggestions. Our research is supported by experiments conducted on 9 large-scale Java systems. The results indicate that although smaller LLMs can perform effectively with appropriate instruction and suitable techniques, there is still considerable potential for improvement in their ability to suggest log levels.

Studying and Benchmarking Large Language Models For Log Level Suggestion

TL;DR

A detailed empirical analysis is undertaken to investigate the impact of characteristics and learning paradigms on the performance of 12 open-source LLMs in log level suggestion, indicating that although smaller LLMs can perform effectively with appropriate instruction and suitable techniques, there is still considerable potential for improvement in their ability to suggest log levels.

Abstract

Large Language Models (LLMs) have become a focal point of research across various domains, including software engineering, where their capabilities are increasingly leveraged. Recent studies have explored the integration of LLMs into software development tools and frameworks, revealing their potential to enhance performance in text and code-related tasks. Log level is a key part of a logging statement that allows software developers control the information recorded during system runtime. Given that log messages often mix natural language with code-like variables, LLMs' language translation abilities could be applied to determine the suitable verbosity level for logging statements. In this paper, we undertake a detailed empirical analysis to investigate the impact of characteristics and learning paradigms on the performance of 12 open-source LLMs in log level suggestion. We opted for open-source models because they enable us to utilize in-house code while effectively protecting sensitive information and maintaining data security. We examine several prompting strategies, including Zero-shot, Few-shot, and fine-tuning techniques, across different LLMs to identify the most effective combinations for accurate log level suggestions. Our research is supported by experiments conducted on 9 large-scale Java systems. The results indicate that although smaller LLMs can perform effectively with appropriate instruction and suitable techniques, there is still considerable potential for improvement in their ability to suggest log levels.

Paper Structure

This paper contains 26 sections, 1 equation, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Logging
  4. Large Language Models
  5. Characteristics of LLMs
  6. Learning Paradigms of LLMs
  7. Privacy Concerns
  8. STUDY DESIGN
  9. Studied Dataset
  10. Selecting Large Language Models (LLMs)
  11. Sampling Few Shot Data
  12. Evaluation Metrics
  13. Results
  14. RQ1: What is the accuracy of LLMs in suggesting log level?
  15. RQ1-A: What is the effectiveness of LLMs in log level suggestion
  16. ...and 11 more sections

Figures (1)

  • Figure 1: Outline of the data processing stage: 1Identify Logging Context: Locate methods with logging statements, 2Collect Logging Information: Determine the log levels and messages, 3Abstract Logging Statements: Remove logging statements and log guards from the source code, 4Construct Prompts: Create prompts for the LLM using the gathered information.