Code Summarization Beyond Function Level
Vladimir Makharev, Vladimir Ivanov
TL;DR
The paper investigates code summarization beyond the function level by introducing class- and repository-level contexts through revised benchmarks (Modified ClassEval and Modified CodeSearchNet) evaluated with both fine-tuned baselines (CodeT5+ variants) and large language models. It systematically studies how context, few-shot learning, and retrieval-augmented generation affect summary quality across function, class, and repository scopes, using six standard metrics and an open-source pipeline. Key findings show that while fine-tuned CodeT5+ often excels at function-level tasks, LLMs can match or beat baselines when provided with high-quality few-shot examples and retrieved context, especially at higher levels; repository-level summarization holds promise but demands substantial compute and structured context. The work provides actionable insights for prompt design, context integration, and benchmark development, and releases datasets and code to support future research in higher-level code summarization.
Abstract
Code summarization is a critical task in natural language processing and software engineering, which aims to generate concise descriptions of source code. Recent advancements have improved the quality of these summaries, enhancing code readability and maintainability. However, the content of a repository or a class has not been considered in function code summarization. This study investigated the effectiveness of code summarization models beyond the function level, exploring the impact of class and repository contexts on the summary quality. The study involved revising benchmarks for evaluating models at class and repository levels, assessing baseline models, and evaluating LLMs with in-context learning to determine the enhancement of summary quality with additional context. The findings revealed that the fine-tuned state-of-the-art CodeT5+ base model excelled in code summarization, while incorporating few-shot learning and retrieved code chunks from RAG significantly enhanced the performance of LLMs in this task. Notably, the Deepseek Coder 1.3B and Starcoder2 15B models demonstrated substantial improvements in metrics such as BLEURT, METEOR, and BLEU-4 at both class and repository levels. Repository-level summarization exhibited promising potential but necessitates significant computational resources and gains from the inclusion of structured context. Lastly, we employed the recent SIDE code summarization metric in our evaluation. This study contributes to refining strategies for prompt engineering, few-shot learning, and RAG, addressing gaps in benchmarks for code summarization at various levels. Finally, we publish all study details, code, datasets, and results of evaluation in the GitHub repository available at https://github.com/kilimanj4r0/code-summarization-beyond-function-level.