Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?

Antonio Vitale, Antonio Mastropaolo, Rocco Oliveto, Massimiliano Di Penta, Simone Scalabrino

TL;DR

This paper investigates whether code-comment coherence, as quantified by the SIDE metric, can optimize code summarization datasets by filtering training instances. By applying SIDE-based thresholds to the TL-CodeSum and Funcom datasets and fine-tuning CodeT5+ on multiple variants, the authors show that training with up to ~50% fewer coherent examples yields performance comparable to using the full dataset, while also reducing training time and energy use. Surprisingly, SIDE-based filtering offers little advantage over random instance selection, suggesting that current datasets already contain high-coherence examples and that other quality attributes should be explored. The work highlights the potential for data-centric optimizations in code summarization but cautions that dataset size alone does not guarantee improvements, urging broader consideration of data diversity and readability to achieve meaningful gains.

Abstract

Automated code summarization is a long-standing goal for code comprehension. This task automatically generates documentation using a given method. Deep Learning (DL)-based approaches have been proven beneficial for various software engineering (SE) tasks, including this one. Most state-of-the-art datasets for code summarization are automatically mined from GitHub and, thus, might contain erroneous or sub-optimal examples. Previous work showed that using a simple rule-based approach for removing noisy instances allows for a tangible reduction of the training set size while not reducing the effectiveness of the trained models. Motivated by this finding, we conjecture that it is possible to further reduce the dataset size by removing instances that contain different issues. In this paper, we explore the extent to which code-comment coherence, a specific quality attribute of code summaries, can be used to optimize code summarization datasets. Specifically, we hypothesize that removing incoherent code-comment pairs might positively impact the effectiveness of the models. To do this, we rely on SIDE, a recently introduced metric for code-summary coherence. We examine multiple selectivity levels of training instances from two state-of-the-art datasets (TL-CodeSum and Funcom) and evaluate the resulting models on three manually curated test sets. The results show that even halving the training set sizes does not significantly affect the model's ability to generate summaries. However, when comparing the most restrictive selection strategy with a simpler one that randomly selects the training instances, we observe that the resulting accuracy of the model also does not change. This result suggests that (i) current datasets contain many irrelevant examples, and (ii) different quality attributes should be explored for optimizing code summarization datasets.

Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?

TL;DR

This paper investigates whether code-comment coherence, as quantified by the SIDE metric, can optimize code summarization datasets by filtering training instances. By applying SIDE-based thresholds to the TL-CodeSum and Funcom datasets and fine-tuning CodeT5+ on multiple variants, the authors show that training with up to ~50% fewer coherent examples yields performance comparable to using the full dataset, while also reducing training time and energy use. Surprisingly, SIDE-based filtering offers little advantage over random instance selection, suggesting that current datasets already contain high-coherence examples and that other quality attributes should be explored. The work highlights the potential for data-centric optimizations in code summarization but cautions that dataset size alone does not guarantee improvements, urging broader consideration of data diversity and readability to achieve meaningful gains.

Abstract

Automated code summarization is a long-standing goal for code comprehension. This task automatically generates documentation using a given method. Deep Learning (DL)-based approaches have been proven beneficial for various software engineering (SE) tasks, including this one. Most state-of-the-art datasets for code summarization are automatically mined from GitHub and, thus, might contain erroneous or sub-optimal examples. Previous work showed that using a simple rule-based approach for removing noisy instances allows for a tangible reduction of the training set size while not reducing the effectiveness of the trained models. Motivated by this finding, we conjecture that it is possible to further reduce the dataset size by removing instances that contain different issues. In this paper, we explore the extent to which code-comment coherence, a specific quality attribute of code summaries, can be used to optimize code summarization datasets. Specifically, we hypothesize that removing incoherent code-comment pairs might positively impact the effectiveness of the models. To do this, we rely on SIDE, a recently introduced metric for code-summary coherence. We examine multiple selectivity levels of training instances from two state-of-the-art datasets (TL-CodeSum and Funcom) and evaluate the resulting models on three manually curated test sets. The results show that even halving the training set sizes does not significantly affect the model's ability to generate summaries. However, when comparing the most restrictive selection strategy with a simpler one that randomly selects the training instances, we observe that the resulting accuracy of the model also does not change. This result suggests that (i) current datasets contain many irrelevant examples, and (ii) different quality attributes should be explored for optimizing code summarization datasets.

Paper Structure

This paper contains 20 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Selection Strategies for Code Summarization
  3. Code-comment cleAning Tool
  4. Fine-Grained Filtering: SIDE
  5. Study Definition, Design and Planning
  6. Context Selection
  7. Study Methodology
  8. Results
  9. RQ$_{0}$ Consistency Analysis
  10. RQ$_{1}$. Selection Strategy Impact
  11. RQ$_2$. Comparison with the Random Baseline
  12. Discussions
  13. Threats to Validity
  14. Related Work
  15. The Importance of Data Quality for DL-Based Methods in Code Summarization
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Example of a $\langle code, summary \rangle$ pair exhibiting a misalignment after the coarse-grained filtering by CAT.
  • Figure 2: Distribution of SIDE scores for TL-CodeSum and Funcom training instances.
  • Figure 3: Pareto front for TL-CodeSum hu2018summarizing (top) and Funcom leclair2019recommendations (bottom). From left to right, the chart shows the number of tokens for $SIDE_{0.9}$, $SIDE_{0.8}$, $SIDE_{0.7}$, $SIDE_{0.6}$, $SIDE_{0.5}$, and the full dataset.