Semantic Similarity Loss for Neural Source Code Summarization

TL;DR

The paper addresses the limitation of word-level loss in neural code summarization by introducing use-seq, a sequence-level semantic similarity loss based on the Universal Sentence Encoder that can be used as a drop-in replacement for categorical cross-entropy (CCE). The approach computes a sentence-level similarity between predicted and reference summaries, broadcasts the score to individual tokens, applies a masking rule, and uses an exponentiated reward controlled by a hyperparameter $\beta$ to combine with CCE. Across multiple purpose-built models and fine-tuned large-language-models, use-seq yields consistent improvements in METEOR, USE, and BLEU on Java and Python datasets, and is supported by a human study showing a preference for use-seq due to higher accuracy and completeness. The work demonstrates practical impact by providing a robust, low-cost enhancement to code summarization training and releasing code and data for reproducibility.

Abstract

This paper presents a procedure for and evaluation of using a semantic similarity metric as a loss function for neural source code summarization. Code summarization is the task of writing natural language descriptions of source code. Neural code summarization refers to automated techniques for generating these descriptions using neural networks. Almost all current approaches involve neural networks as either standalone models or as part of a pretrained large language models e.g., GPT, Codex, LLaMA. Yet almost all also use a categorical cross-entropy (CCE) loss function for network optimization. Two problems with CCE are that 1) it computes loss over each word prediction one-at-a-time, rather than evaluating a whole sentence, and 2) it requires a perfect prediction, leaving no room for partial credit for synonyms. In this paper, we extend our previous work on semantic similarity metrics to show a procedure for using semantic similarity as a loss function to alleviate this problem, and we evaluate this procedure in several settings in both metrics-driven and human studies. In essence, we propose to use a semantic similarity metric to calculate loss over the whole output sentence prediction per training batch, rather than just loss for each word. We also propose to combine our loss with CCE for each word, which streamlines the training process compared to baselines. We evaluate our approach over several baselines and report improvement in the vast majority of conditions.

Figures (3)

• Figure 1: Screenshot of page one of the survey interface.
• Figure 2: Aggregate survey responses. Each column indicates the total number of survey replies for a quality attribute and summaries from one loss function. With 29 participants and 56 functions, there are 1624 responses per attribute. For example, there were $\sim$900 responses in which participants preferred use-seq in terms of accuracy versus $\sim$550 for CCE.
• Figure 3: Table shows a statistical summary of responses for use-seq grouped by participant, followed by Friedman significance test results. For example, an "average" participant, in terms of accuracy, preferred 30.58 summaries by use-seq compared to $\sim$19 for CCE, and had no preference for $\sim$6. Boxplots show comparisons grouped by participant for all questions.