A Prompt Learning Framework for Source Code Summarization

TL;DR

This work introduces PromptCS, a prompt learning framework for source code summarization that trains a prompt agent to generate continuous prompts while keeping large language model (LLM) parameters frozen. The prompt agent, guided by the LLM, yields continuous prompts that better induce code summarization, delivering performance that surpasses instruction prompting (zero/few-shot) and rivals task-oriented fine-tuning, with notable training efficiency especially for larger LLMs. Through CodeSearchNet-based evaluation across multiple languages and LLMs, PromptCS demonstrates strong automated and human-evaluation results, generalizes to additional programming languages, and requires significantly less training data and compute than full fine-tuning. The approach offers a non-invasive, reusable mechanism to adapt LLMs to code-related tasks, and the authors discuss broader implications for PEFT-informed, prompt-based adaptation in SE tasks.

Abstract

(Source) code summarization is the task of automatically generating natural language summaries (also called comments) for given code snippets. Recently, with the successful application of large language models (LLMs) in numerous fields, software engineering researchers have also attempted to adapt LLMs to solve code summarization tasks. The main adaptation schemes include instruction prompting, task-oriented (full-parameter) fine-tuning, and parameter-efficient fine-tuning (PEFT). However, instruction prompting involves designing crafted prompts and requires users to have professional domain knowledge, while task-oriented fine-tuning requires high training costs, and effective, tailored PEFT methods for code summarization are still lacking. This paper proposes an effective prompt learning framework for code summarization called PromptCS. It no longer requires users to rack their brains to design effective prompts. Instead, PromptCS trains a prompt agent that can generate continuous prompts to unleash the potential for LLMs in code summarization. Compared to the human-written discrete prompt, the continuous prompts are produced under the guidance of LLMs and are therefore easier to understand by LLMs. PromptCS is non-invasive to LLMs and freezes the parameters of LLMs when training the prompt agent, which can greatly reduce the requirements for training resources. Our comprehensive experimental results show that PromptCS significantly outperforms instruction prompting schemes (including zero-shot learning and few-shot learning) on all four widely used metrics, and is comparable to the task-oriented fine-tuning scheme. In some base LLMs, e.g., StarCoderBase-1B and -3B, PromptCS even outperforms the task-oriented fine-tuning scheme. More importantly, the training efficiency of PromptCS is faster than the task-oriented fine-tuning scheme, with a more pronounced advantage on larger LLMs.

Figures (8)

• Figure 1: An example of a code snippet and its summary
• Figure 2: Overview of PromptCS
• Figure 3: Example of prompt and code concatenation
• Figure 4: Training costs for different prompt lengths
• Figure 5: Influence of training set size on task-oriented fine-tuning