Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Quality of AI-Generated Source Code Comments: A Comprehensive Evaluation

Ian Guelman, Arthur Gregório Leal, Laerte Xavier, Marco Tulio Valente

TL;DR

The paper conducts a large-scale, multi-model evaluation of AI-generated Java Javadocs, combining qualitative human judgments with quantitative IR metrics and CK-based code properties. It finds that a majority of generated comments are equivalent to or better than originals, while conventional metrics like BLEU poorly reflect human judgments; ROUGE-L and METEOR align better but still miss nuances, underscoring the need for new, documentation-focused metrics. The results suggest that richer code context aids generation and support practical tooling for automated documentation, accompanied by a robust replication package. Overall, the work provides a nuanced baseline for assessing AI-assisted code documentation and outlines concrete directions for metric development and tooling integration.

Abstract

This paper investigates the quality of source code comments automatically generated by Large Language Models (LLMs). While AI-based comment generation has emerged as a promising solution to reduce developers' documentation effort, prior studies have been limited by small datasets or by relying solely on traditional Information Retrieval (IR) metrics, which are insufficient to capture documentation quality. To address these limitations, we conducted a large-scale empirical study on 142 classes and 273 methods created after the training cut-off of the evaluated models. For each code element, we generated Javadoc comments using three LLMs (GPT-3.5 Turbo, GPT-4o, and DeepSeek-V3). A qualitative assessment of the comments-performed independently by two experts-showed that 58.8% were equivalent to, and 27.7% superior to, the original comments. A quantitative analysis using BLEU, ROUGE-L, and METEOR confirmed that IR-based metrics do not reliably reflect human evaluations, revealing the need for new documentation-specific metrics. Finally, correlation analyses indicated slightly positive relationships between code properties (size, complexity, coupling) and comment quality, confirming that LLMs benefit from richer contextual information.

On the Quality of AI-Generated Source Code Comments: A Comprehensive Evaluation

TL;DR

The paper conducts a large-scale, multi-model evaluation of AI-generated Java Javadocs, combining qualitative human judgments with quantitative IR metrics and CK-based code properties. It finds that a majority of generated comments are equivalent to or better than originals, while conventional metrics like BLEU poorly reflect human judgments; ROUGE-L and METEOR align better but still miss nuances, underscoring the need for new, documentation-focused metrics. The results suggest that richer code context aids generation and support practical tooling for automated documentation, accompanied by a robust replication package. Overall, the work provides a nuanced baseline for assessing AI-assisted code documentation and outlines concrete directions for metric development and tooling integration.

Abstract

This paper investigates the quality of source code comments automatically generated by Large Language Models (LLMs). While AI-based comment generation has emerged as a promising solution to reduce developers' documentation effort, prior studies have been limited by small datasets or by relying solely on traditional Information Retrieval (IR) metrics, which are insufficient to capture documentation quality. To address these limitations, we conducted a large-scale empirical study on 142 classes and 273 methods created after the training cut-off of the evaluated models. For each code element, we generated Javadoc comments using three LLMs (GPT-3.5 Turbo, GPT-4o, and DeepSeek-V3). A qualitative assessment of the comments-performed independently by two experts-showed that 58.8% were equivalent to, and 27.7% superior to, the original comments. A quantitative analysis using BLEU, ROUGE-L, and METEOR confirmed that IR-based metrics do not reliably reflect human evaluations, revealing the need for new documentation-specific metrics. Finally, correlation analyses indicated slightly positive relationships between code properties (size, complexity, coupling) and comment quality, confirming that LLMs benefit from richer contextual information.
Paper Structure (17 sections, 12 figures)

This paper contains 17 sections, 12 figures.

Table of Contents

  1. Introduction
  2. Study Design
  3. Dataset
  4. Ground Truth
  5. Javadoc Generation Using LLMs
  6. Qualitative Analysis
  7. Quantitative Analysis
  8. Source Code Metrics Analysis
  9. Metadata
  10. Qualitative Results
  11. Human Assessment Scoring Examples
  12. Quantitative Results
  13. Qualitative vs Quantitative Results
  14. Source Code Metrics Analysis
  15. Threats to Validity
  16. ...and 2 more sections

Figures (12)

  • Figure 1: Prompt structure
  • Figure 2: Human assessment scores
  • Figure 3: Human Assessment Score---Example Score 4+
  • Figure 4: Human Assessment Score---Example Score 4
  • Figure 5: Human Assessment Score---Example Score 3
  • ...and 7 more figures