AI-Mediated Code Comment Improvement

TL;DR

This work identifies seven quality axes for code comments through a grounded-theory study of programmer rationales and then presents an AI-mediated pipeline to rewrite comments along those axes. The pipeline uses both GPT-4o (with RLAIF) and an in-house distilled model to generate and refine axis-aligned summaries, followed by fine-tuning per axis, and evaluation against human references. Empirical results show substantial improvements in alignment with human-selected comments, particularly for the larger CodeLlama-based system, with publicly released data and code to support reproducibility. The approach enables in-house, data-custody-preserving deployment of improved code comments across Java methods, potentially enhancing maintainability and comprehension at scale.

Abstract

This paper describes an approach to improve code comments along different quality axes by rewriting those comments with customized Artificial Intelligence (AI)-based tools. We conduct an empirical study followed by grounded theory qualitative analysis to determine the quality axes to improve. Then we propose a procedure using a Large Language Model (LLM) to rewrite existing code comments along the quality axes. We implement our procedure using GPT-4o, then distil the results into a smaller model capable of being run in-house, so users can maintain data custody. We evaluate both our approach using GPT-4o and the distilled model versions. We show in an evaluation how our procedure improves code comments along the quality axes. We release all data and source code in an online repository for reproducibility.

Figures (5)

• Figure 1: First page of the study interface.
• Figure 2: Second page of the study interface.
• Figure 3: Distribution of tags from our annotation process.
• Figure 4: An overview of the three steps in our approach and their inputs.
• Figure 5: Prompt template we used for the first-pass LLM-based improvement for the "precise" quality axis. We replace {summaries[x]} and {code} tags with a summary candidate ($0\leq x\leq n$, where $n$ is the number of candidates) and the method's Java source code.