Automated Generation of Commit Messages in Software Repositories

TL;DR

This paper tackles automated generation of commit messages from code diffs to improve documentation quality under limited developer time. It evaluates lightweight ML/NLP approaches—TF-IDF with cosine similarity, Word2Vec with Logistic Regression, and an LSTM—on a dataset of code changes and commit messages derived from prior work. The study finds that TF-IDF-based cosine similarity with nearest neighbors achieves the strongest BLEU score of 16.82, while LSTM underperforms, with LLMs like ChatGPT performing best for large diffs but lacking local deployability. The results provide practical insights into efficient, offline commit message generation and invite future work on broader languages, pre-trained models, and concise messaging.

Abstract

Commit messages are crucial for documenting software changes, aiding in program comprehension and maintenance. However, creating effective commit messages is often overlooked by developers due to time constraints and varying levels of documentation skills. Our research presents an automated approach to generate commit messages using Machine Learning (ML) and Natural Language Processing (NLP) by developing models that use techniques such as Logistic Regression with TF-IDF and Word2Vec, as well as more sophisticated methods like LSTM. We used the dataset of code changes and corresponding commit messages that was used by Liu et al., which we used to train and evaluate ML/NLP models and was chosen because it is extensively used in previous research, also for comparability in our study. The objective was to explore which ML/NLP techniques generate the most effective, clear, and concise commit messages that accurately reflect the code changes. We split the dataset into training, validation, and testing sets and used these sets to evaluate the performance of each model using qualitative and quantitative evaluation methods. Our results reveal a spectrum of effectiveness among these models, with the highest BLEU score achieved being 16.82, showcasing the models' capability in automating a clear and concise commit message generation. Our paper offers insights into the comparative effectiveness of different machine learning models for automating commit message generation in software development, aiming to enhance the overall practice of code documentation. The source code is available at https://doi.org/10.5281/zenodo.10888106.

Figures (1)

• Figure 1: The architecture of the NLP model with the LSTM approach