Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models

TL;DR

This work addresses the limitation of generating a single code summary by introducing Variational Prefix Tuning (VPT), a parameter-efficient method that integrates a Conditional VAE into pre-trained transformer models to produce a diverse set of accurate code summaries. By sampling latent variables from a learned posterior and using them as prefixes during decoding, VPT expands the output space without retraining large models, and a bi-criteria subset selection balances quality and diversity. Empirical results on Java and Python datasets across multiple backbones (including CodeT5+, NeuralCodeSum, SCRIPT, PLBART, and CodeLlama with LoRA) show that VPT improves oracle-based accuracy and yields competitive or superior diversity compared with standard diverse decoding baselines and even GPT-4o in limited comparisons. The approach demonstrates strong cross-model adaptability, substantial parameter efficiency, and practical potential for providing developers with a rich set of viable code summaries, along with a replication package for reproducibility.

Abstract

Recent advancements in source code summarization have leveraged transformer-based pre-trained models, including Large Language Models of Code (LLMCs), to automate and improve the generation of code summaries. However, existing methods often focus on generating a single high-quality summary for a given source code, neglecting scenarios where the generated summary might be inadequate and alternative options are needed. In this paper, we introduce Variational Prefix Tuning (VPT), a novel approach that enhances pre-trained models' ability to generate diverse yet accurate sets of summaries, allowing the user to choose the most suitable one for the given source code. Our method integrates a Conditional Variational Autoencoder (CVAE) framework as a modular component into pre-trained models, enabling us to model the distribution of observed target summaries and sample continuous embeddings to be used as prefixes to steer the generation of diverse outputs during decoding. Importantly, we construct our method in a parameter-efficient manner, eliminating the need for expensive model retraining, especially when using LLMCs. Furthermore, we employ a bi-criteria reranking method to select a subset of generated summaries, optimizing both the diversity and the accuracy of the options presented to users. We present extensive experimental evaluations using widely used datasets and current state-of-the-art pre-trained code summarization models to demonstrate the effectiveness of our approach and its adaptability across models.

Figures (2)

• Figure 1: An overview for using VPT to generate a diverse and accurate set of code summaries.
• Figure 2: Detailed Model Architecture of VPT.