Distilled GPT for Source Code Summarization

TL;DR

Code summaries are essential for developer documentation but API-based solutions raise data-custody concerns. The authors distill GPT-3.5's code-summarization ability into open-source, smaller models using 2.15M GPT-3.5-generated samples and fine-tune decoder-only and encoder-decoder architectures to enable on-device inference on a single 16GB GPU. They demonstrate that GPT-3.5 summaries outperform human references in accuracy and completeness, and they conduct a comprehensive distillation study across model and data sizes, followed by human evaluations showing the distilled model closely mimics GPT-3.5. The work also emphasizes reproducibility by releasing data and code, highlighting practical custody-preserving deployment with a 350M-parameter model as a cost-effective alternative to runaway-scale LLMs for code summarization.

Abstract

A code summary is a brief natural language description of source code. Summaries are usually only a single sentence long, and yet form the backbone of developer documentation. A short descriptions such as "changes all visible polygons to the color blue" can give a programmer a high-level idea of what code does without the effort of reading the code itself. Recently, products based on Large Language Models such as ChatGPT have demonstrated a strong ability to write these descriptions automatically. However, to use these tools, programmers must send their code to untrusted third parties for processing (e.g., via an API call). This loss of custody is not acceptable to many organizations. In this paper, we present an alternative: we train an open source model using sample output generated by GPT-3.5 in a process related to knowledge distillation. Our model is small enough (350m parameters) to be run on a single 16gb GPU, yet we show in our evaluation that it is large enough to mimic GPT-3.5 on this task.

Figures (6)

• Figure 1: Example of the two pages of our survey for each Java method.
• Figure 2: Comparison of human reference summaries to GPT-3.5, along the four questions we discuss in Section \ref{['sec:rq1method']}, followed by a statistical summary of the RQ1 results in Figure \ref{['fig:rq1results']} with Mann-Whitney tests showing statistical significance.
• Figure 3: Comparison of human reference summaries to GPT-3.5 in terms of overall preference and the number of convergence and divergence functions on the duplicate studies
• Figure 4: Comparison of summaries generated by models trained with human reference summaries to summaries generated by models trained with GPT-3.5, along the four questions we discuss in Section \ref{['sec:rq1method']}, followed by a statistical summary of the RQ1 results in Figure \ref{['fig:rq1resultstrained']} with Mann-Whitney tests.
• Figure 5: Comparison of jam to GPT-3.5 followed by a statistical summary of the RQ3 results with Mann-Whitney tests showing statistical significance.