Commenting Higher-level Code Unit: Full Code, Reduced Code, or Hierarchical Code Summarization

TL;DR

The paper addresses the gap in higher-level code summarization by systematically studying file-level and module-level ACS using large language models. It compares three summarization strategies—full code, reduced code, and hierarchical summarization—across CodeLlama, CodeGemma, and GPT-4, on 10 Java projects. Key findings show that full code is best for file-level summaries, while hierarchical summarization (HMCS) is most effective for module-level summaries, with reduced code offering cost-efficient alternatives under budget constraints. Additionally, the study demonstrates that LLMs can serve as approximate evaluators for summary quality, with GPT-4 showing strong correlation to human judgments, and it provides public datasets and code to support replication and future benchmarking.

Abstract

Commenting code is a crucial activity in software development, as it aids in facilitating future maintenance and updates. To enhance the efficiency of writing comments and reduce developers' workload, researchers has proposed various automated code summarization (ACS) techniques to automatically generate comments/summaries for given code units. However, these ACS techniques primarily focus on generating summaries for code units at the method level. There is a significant lack of research on summarizing higher-level code units, such as file-level and module-level code units, despite the fact that summaries of these higher-level code units are highly useful for quickly gaining a macro-level understanding of software components and architecture. To fill this gap, in this paper, we conduct a systematic study on how to use LLMs for commenting higher-level code units, including file level and module level. These higher-level units are significantly larger than method-level ones, which poses challenges in handling long code inputs within LLM constraints and maintaining efficiency. To address these issues, we explore various summarization strategies for ACS of higher-level code units, which can be divided into three types: full code summarization, reduced code summarization, and hierarchical code summarization. The experimental results suggest that for summarizing file-level code units, using the full code is the most effective approach, with reduced code serving as a cost-efficient alternative. However, for summarizing module-level code units, hierarchical code summarization becomes the most promising strategy. In addition, inspired by the research on method-level ACS, we also investigate using the LLM as an evaluator to evaluate the quality of summaries of higher-level code units. The experimental results demonstrate that the LLM's evaluation results strongly correlate with human evaluations.

Figures (11)

• Figure 1: An example of ACS on different level code units.
• Figure 2: An example of the FFCS strategy.
• Figure 3: An example of reduced file code.
• Figure 4: An example of the RFCS strategy.
• Figure 5: An example of the HFCS_m strategy.