Few-shot training LLMs for project-specific code-summarization
Toufique Ahmed, Premkumar Devanbu
TL;DR
The study examines few-shot training of Codex for project-specific code summarization, comparing cross-project and same-project prompts against state-of-the-art fine-tuned baselines on CodeX-GLUE. Using a 10-shot prompt framework, Codex consistently improves BLEU-4 scores over baselines, with same-project prompts delivering additional gains and statistical significance. Zero-shot and one-shot attempts perform poorly, highlighting the value of targeted few-shot conditioning. The findings suggest practical, data-efficient strategies for generating accurate code comments tailored to a project's vocabulary and conventions, with potential applicability to other software engineering tasks.
Abstract
Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.