Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Large Language Models for Commit Message Generation: A Preliminary Study

Linghao Zhang, Jingshu Zhao, Chong Wang, Peng Liang

TL;DR

This study investigates whether large language models can automatically generate informative commit messages. It evaluates two LLMs, ChatGPT and Llama 2, against established baselines using a two-phase framework that combines automatic metrics (BLEU variants and Rouge-L) with human judgments on a dataset of 7,661 code-diff/commit-message pairs. Results show LLMs deliver competitive automatic-metric performance and superior performance in human evaluation, with 78% of cases rated as best when generated by LLMs, highlighting both potential and the limitations of current metrics. The work suggests directions for improved prompting and practical integration of LLM-powered commitMessage generation into software development workflows.

Abstract

A commit message is a textual description of the code changes in a commit, which is a key part of the Git version control system (VCS). It captures the essence of software updating. Therefore, it can help developers understand code evolution and facilitate efficient collaboration between developers. However, it is time-consuming and labor-intensive to write good and valuable commit messages. Some researchers have conducted extensive studies on the automatic generation of commit messages and proposed several methods for this purpose, such as generationbased and retrieval-based models. However, seldom studies explored whether large language models (LLMs) can be used to generate commit messages automatically and effectively. To this end, this paper designed and conducted a series of experiments to comprehensively evaluate the performance of popular open-source and closed-source LLMs, i.e., Llama 2 and ChatGPT, in commit message generation. The results indicate that considering the BLEU and Rouge-L metrics, LLMs surpass the existing methods in certain indicators but lag behind in others. After human evaluations, however, LLMs show a distinct advantage over all these existing methods. Especially, in 78% of the 366 samples, the commit messages generated by LLMs were evaluated by humans as the best. This work not only reveals the promising potential of using LLMs to generate commit messages, but also explores the limitations of commonly used metrics in evaluating the quality of auto-generated commit messages.

Using Large Language Models for Commit Message Generation: A Preliminary Study

TL;DR

This study investigates whether large language models can automatically generate informative commit messages. It evaluates two LLMs, ChatGPT and Llama 2, against established baselines using a two-phase framework that combines automatic metrics (BLEU variants and Rouge-L) with human judgments on a dataset of 7,661 code-diff/commit-message pairs. Results show LLMs deliver competitive automatic-metric performance and superior performance in human evaluation, with 78% of cases rated as best when generated by LLMs, highlighting both potential and the limitations of current metrics. The work suggests directions for improved prompting and practical integration of LLM-powered commitMessage generation into software development workflows.

Abstract

A commit message is a textual description of the code changes in a commit, which is a key part of the Git version control system (VCS). It captures the essence of software updating. Therefore, it can help developers understand code evolution and facilitate efficient collaboration between developers. However, it is time-consuming and labor-intensive to write good and valuable commit messages. Some researchers have conducted extensive studies on the automatic generation of commit messages and proposed several methods for this purpose, such as generationbased and retrieval-based models. However, seldom studies explored whether large language models (LLMs) can be used to generate commit messages automatically and effectively. To this end, this paper designed and conducted a series of experiments to comprehensively evaluate the performance of popular open-source and closed-source LLMs, i.e., Llama 2 and ChatGPT, in commit message generation. The results indicate that considering the BLEU and Rouge-L metrics, LLMs surpass the existing methods in certain indicators but lag behind in others. After human evaluations, however, LLMs show a distinct advantage over all these existing methods. Especially, in 78% of the 366 samples, the commit messages generated by LLMs were evaluated by humans as the best. This work not only reveals the promising potential of using LLMs to generate commit messages, but also explores the limitations of commonly used metrics in evaluating the quality of auto-generated commit messages.
Paper Structure (15 sections, 3 figures, 2 tables)

This paper contains 15 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Research Design
  4. Research Question
  5. Overview of Our Approach
  6. Dataset
  7. Selection and Settings of LLMs
  8. Metrics and Baselines used in Evaluation: Phase I
  9. Human Evaluation: Phase II
  10. Results & Discussion
  11. Result of Phase I: Metrics evaluation
  12. Result of Phase II: Human evaluation
  13. Discussion
  14. Limitations
  15. Conclusions & Future Work

Figures (3)

  • Figure 1: Overview of our approach
  • Figure 2: Example of human evaluation
  • Figure 3: Human evaluation results (number of times selected as the best commit message corresponding to the code diff - average value of two participants)