An Empirical Study on Commit Message Generation using LLMs via In-Context Learning

TL;DR

The paper investigates commit message generation using large language models through in-context learning, evaluating how prompts, demonstrations, and retrieval-based selection affect performance on a multilingual benchmark (MCMD) and a leakage-mitigated dataset (MCMD-New). It finds that well-designed prompts and a moderate number of demonstrations, especially with retrieval-based selection, enable LLMs (notably GPT-3.5-Turbo and DeepSeek-V2-Chat) to outperform state-of-the-art baselines on generalization tasks and to align closely with human judgments in subjective evaluation. The study highlights the importance of high-quality datasets and LLM-based evaluators, and it discusses root causes of underperformance, offering directions for improving LLM-assisted commit message generation. Overall, the work demonstrates the feasibility and practicality of using ICL with LLMs to generate commit messages without model fine-tuning, while calling for further research into dataset quality and domain-specific language models.

Abstract

Commit messages concisely describe code changes in natural language and are important for software maintenance. Several approaches have been proposed to automatically generate commit messages, but they still suffer from critical limitations, such as time-consuming training and poor generalization ability. To tackle these limitations, we propose to borrow the weapon of large language models (LLMs) and in-context learning (ICL). Our intuition is based on the fact that the training corpora of LLMs contain extensive code changes and their pairwise commit messages, which makes LLMs capture the knowledge about commits, while ICL can exploit the knowledge hidden in the LLMs and enable them to perform downstream tasks without model tuning. However, it remains unclear how well LLMs perform on commit message generation via ICL. In this paper, we conduct an empirical study to investigate the capability of LLMs to generate commit messages via ICL. Specifically, we first explore the impact of different settings on the performance of ICL-based commit message generation. We then compare ICL-based commit message generation with state-of-the-art approaches on a popular multilingual dataset and a new dataset we created to mitigate potential data leakage. The results show that ICL-based commit message generation significantly outperforms state-of-the-art approaches on subjective evaluation and achieves better generalization ability. We further analyze the root causes for LLM's underperformance and propose several implications, which shed light on future research directions for using LLMs to generate commit messages.

Figures (9)

• Figure 1: An example of a commit with a code diff and its reference message.
• Figure 2: Overview of our study.
• Figure 3: Overview of ICL-based commit message generation.
• Figure 4: The prompt for LLM-based evaluation.
• Figure 5: Impact of different prompts and demonstration number on the performance of ChatGPT.