COMCAT: Leveraging Human Judgment to Improve Automatic Documentation and Summarization

TL;DR

COMCAT introduces a context-aware approach to automatic code commenting by augmenting LLMs with developer-guided context. It deploys a five-component pipeline (Code Parser, Code Classifier, Template Catalog, Template-Based Prompter, ChatGPT) to generate inline and function-level comments for C/C++ code, guided by three human-subject studies. Results show up to a $12\%$ improvement in developer comprehension for $87\%$ of participants, with comments that are at least as accurate and readable as human-written ones and often preferred over standard ChatGPT output. A dataset of code snippets, human-written comments, and annotated categories is released to support future research.

Abstract

Software maintenance constitutes a substantial portion of the total lifetime costs of software, with a significant portion attributed to code comprehension. Software comprehension is eased by documentation such as comments that summarize and explain code. We present COMCAT, an approach to automate comment generation by augmenting Large Language Models (LLMs) with expertise-guided context to target the annotation of source code with comments that improve comprehension. Our approach enables the selection of the most relevant and informative comments for a given snippet or file containing source code. We develop the COMCAT pipeline to comment C/C++ files by (1) automatically identifying suitable locations in which to place comments, (2) predicting the most helpful type of comment for each location, and (3) generating a comment based on the selected location and comment type. In a human subject evaluation, we demonstrate that COMCAT-generated comments significantly improve developer code comprehension across three indicative software engineering tasks by up to 12% for 87% of participants. In addition, we demonstrate that COMCAT-generated comments are at least as accurate and readable as human-generated comments and are preferred over standard ChatGPT-generated comments for up to 92% of snippets of code. Furthermore, we develop and release a dataset containing source code snippets, human-written comments, and human-annotated comment categories. COMCAT leverages LLMs to offer a significant improvement in code comprehension across a variety of human software engineering tasks.

Figures (3)

• Figure 1: ComCat pipeline and study procedure. We use three instances of human subjects research to inform ComCat's design (1) and evaluate developer performance (2) and preference (3) with our tool. ComCat takes C/C++ code as input, using a Code Parser to identify code Snippets to be commented. These Snippets are classified, and the class of each Snippet is used in combination with our Template Catalog to create a prompt for each Snippet. These prompt ChatGPT, which outputs the commented code. This pipeline is informed by developer expertise, but it is fully automated and requires no human intervention.
• Figure 2: Results from HSR2. This shows the difference in correctness between ComCat-generated and human-written comments for each question type.
• Figure 3: Density of comment types in our dataset. Here, density is the number of comments per 100 LOC. ComCat improves comment density over humans, addressing the problem of sparsely commented, minimally maintainable code without cluttering code with excessive documentation.