CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation
Bo Xiong, Linghao Zhang, Zongen Ren, Chong Wang, Peng Liang
TL;DR
CoRaCMG tackles the CMG problem by augmenting LLM inputs with retrieved exemplar diff–message pairs, enabling learning of project-specific terminology and writing style without fine-tuning. The framework combines a hybrid lexical-semantic retriever with a structured prompt and performs generation in three phases: Retrieve, Augment, Generate. Across multiple LLMs and languages, CoRaCMG yields substantial improvements on BLEU, Rouge-L, METEOR, and CIDEr, with single exemplars delivering large gains and diminishing returns beyond three exemplars. The approach demonstrates strong cross-language generalizability and practical utility, offering a training-free, adaptable method to boost commit message quality in real-world software engineering contexts.
Abstract
Commit messages play a key role in documenting the intent behind code changes. However, they are often low-quality, vague, or incomplete, limiting their usefulness. Commit Message Generation (CMG) aims to automatically generate descriptive commit messages from code diffs to reduce developers' effort and improve message quality. Although recent advances in LLMs have shown promise in automating CMG, their performance remains limited. This paper aims to enhance CMG performance by retrieving similar diff-message pairs to guide LLMs to generate commit messages that are more precise and informative. We proposed CoRaCMG, a Contextual Retrieval-augmented framework for Commit Message Generation, structured in three phases: (1) Retrieve: retrieving the similar diff-message pairs; (2) Augment: combining them with the query diff into a structured prompt; and (3) Generate: generating commit messages corresponding to the query diff via LLMs. CoRaCMG enables LLMs to learn project-specific terminologies and writing styles from the retrieved diff-message pairs. We evaluated CoRaCMG across multiple LLMs (e.g., GPT, DeepSeek, and Qwen) and compared its performance against SOTA baselines. Experimental results show that CoRaCMG significantly boosts LLM performance across four metrics (BLEU, Rouge-L, METEOR, and CIDEr). Specifically, DeepSeek-R1 achieves relative improvements of 76% in BLEU and 71% in CIDEr when augmented with a single retrieved example pair. After incorporating the single example pair, GPT-4o achieves the highest improvement rate, with BLEU increasing by 89%. Moreover, performance gains plateau after more than three examples are used, indicating diminishing returns. Further analysis shows that the improvements are attributed to the model's ability to capture the terminologies and writing styles of human-written commit messages from the retrieved example pairs.