Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAG-Enhanced Commit Message Generation

Linghao Zhang, Hongyi Zhang, Chong Wang, Peng Liang

TL;DR

This paper addresses the challenge of generating high-quality commit messages by introducing REACT, a retrieval-augmented framework that combines a hybrid retriever, input augmentation, and generation with Code Language Models. REACT retrieves an exemplar diff–message pair, augments the query input with this exemplar, and generates the commit message using either PLMs (with fine-tuning) or LLMs (via in-context prompting). Extensive experiments across multiple CLMs and LLMs show substantial improvements over baselines, with CodeT5 achieving a BLEU increase of up to 55% and Llama 3 up to 102% when integrated into REACT, and a within-project case study demonstrating adaptation to project-specific conventions. The work demonstrates the practical value of retrieval-augmented generation for CMG and provides a robust replication setup and broader implications for applying RAG in software engineering tasks.

Abstract

Commit message is one of the most important textual information in software development and maintenance. However, it is time-consuming to write commit messages manually. Commit Message Generation (CMG) has become a research hotspot. Recently, several pre-trained language models (PLMs) and large language models (LLMs) with code capabilities have been introduced, demonstrating impressive performance on code-related tasks. Meanwhile, prior studies have explored the utilization of retrieval techniques for CMG, but it is still unclear what effects would emerge from combining advanced retrieval techniques with various generation models. This paper proposed REACT, a REtrieval-Augmented framework for CommiT message generation. It integrates advanced retrieval techniques with different PLMs and LLMs, to enhance the performance of these models on the CMG task. Specifically, a hybrid retriever is designed and used to retrieve the most relevant code diff and commit message pair as an exemplar. Then, the retrieved pair is utilized to guide and enhance the CMG task by PLMs and LLMs through fine-tuning and in-context learning. The experimental results show that REACT significantly enhances these models' performance on the CMG task, improving the BLEU score of CodeT5 by up to 55%, boosting Llama 3's BLEU score by 102%, and substantially surpassing all baselines.

RAG-Enhanced Commit Message Generation

TL;DR

This paper addresses the challenge of generating high-quality commit messages by introducing REACT, a retrieval-augmented framework that combines a hybrid retriever, input augmentation, and generation with Code Language Models. REACT retrieves an exemplar diff–message pair, augments the query input with this exemplar, and generates the commit message using either PLMs (with fine-tuning) or LLMs (via in-context prompting). Extensive experiments across multiple CLMs and LLMs show substantial improvements over baselines, with CodeT5 achieving a BLEU increase of up to 55% and Llama 3 up to 102% when integrated into REACT, and a within-project case study demonstrating adaptation to project-specific conventions. The work demonstrates the practical value of retrieval-augmented generation for CMG and provides a robust replication setup and broader implications for applying RAG in software engineering tasks.

Abstract

Commit message is one of the most important textual information in software development and maintenance. However, it is time-consuming to write commit messages manually. Commit Message Generation (CMG) has become a research hotspot. Recently, several pre-trained language models (PLMs) and large language models (LLMs) with code capabilities have been introduced, demonstrating impressive performance on code-related tasks. Meanwhile, prior studies have explored the utilization of retrieval techniques for CMG, but it is still unclear what effects would emerge from combining advanced retrieval techniques with various generation models. This paper proposed REACT, a REtrieval-Augmented framework for CommiT message generation. It integrates advanced retrieval techniques with different PLMs and LLMs, to enhance the performance of these models on the CMG task. Specifically, a hybrid retriever is designed and used to retrieve the most relevant code diff and commit message pair as an exemplar. Then, the retrieved pair is utilized to guide and enhance the CMG task by PLMs and LLMs through fine-tuning and in-context learning. The experimental results show that REACT significantly enhances these models' performance on the CMG task, improving the BLEU score of CodeT5 by up to 55%, boosting Llama 3's BLEU score by 102%, and substantially surpassing all baselines.
Paper Structure (58 sections, 1 equation, 6 figures, 6 tables)

This paper contains 58 sections, 1 equation, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Motivation.
  3. Our approach.
  4. Results.
  5. Contributions.
  6. Related Work
  7. Commit Message Generation
  8. Retrieval-based methods
  9. Learning-based methods
  10. Hybrid methods
  11. Code Language Models
  12. Pre-trained Language Models (PLMs) for Code
  13. Large Language Models (LLMs)
  14. Retrieval-Augmented Generation
  15. Methodology
  16. ...and 43 more sections

Figures (6)

  • Figure 1: A motivating scenario of how developers write a commit message by referring to a similar exemplar.
  • Figure 2: Overview of our REACT framework. Regarding a code diff (query diff) awaiting the generation of its corresponding commit message, we first retrieve a relevant diff-message pair from the source database using the hybrid retriever. After that, we put query diff and retrieved pair together into the augmenter for concatenation or filling the prompt template. The alterable generator receives formatted inputs and generates a commit message under the guidance of the retrieved pair.
  • Figure 3: Direct prompt template and REACT prompt template for LLMs.
  • Figure 4: An example of CLMs generating commit message according to a code diff.
  • Figure 5: A case of generating commit message using Llama 3 integrated with REACT.
  • ...and 1 more figures