Operationalizing Large Language Models with Design-Aware Contexts for Code Comment Generation

TL;DR

This work addresses the challenge of generating high-quality explanatory code comments for novice maintainers by using design documents as project-specific context in a Retrieval-Augmented Generation framework. It introduces a pipeline that combines source code, AST information, design documents, and RAG to produce docstring-like comments for C/C++ code and evaluates them with automated metrics and a human study. The results indicate that design-document context improves comment usefulness and reduces maintenance time, while abstract syntax trees can overwhelm models and worsen quality; OpenAI's o3 variant provides the best bug-fixing performance. The work offers a practical approach to enhancing maintainability in large, novice-heavy codebases and points to IDE integration and more cost-effective models as future directions.

Abstract

Comments are very useful to the flow of code development. With the increasing commonality of code, novice coders have been creating a significant amount of codebases. Due to lack of commenting standards, their comments are often useless, and increase the time taken to further maintain codes. This study intends to find the usefulness of large language models (LLMs) in these cases to generate potentially better comments. This study focuses on the feasibility of design documents as a context for the LLMs to generate more useful comments, as design documents are often used by maintainers to understand code when comments do not suffice.

Figures (2)

• Figure 1: The lines across the violins are the quartiles. The brighter points are the cases with the design documents in the context, and the dimmer points are the cases without it. The left two violins are without the AST, the right two are with it.
• Figure 2: Decrease in completeness with increasing file size, showing deteriorating performance of the LLMs