Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-quality data augmentation for code comment classification

Thomas Borsani, Andrea Rosani, Giuseppe Di Fatta

TL;DR

Code comment classification suffers from small, imbalanced datasets across languages in the NLBSE'26 benchmark. The authors propose Q-SYNTH, a three-module framework that generates diverse samples with a BERT-like generator, assesses semantic quality via a sentence-transformer, and selects high-quality samples to balance classes, integrated with a multi-task learning head using the SAMGS optimizer. Their results show that high-quality synthetic data can outperform larger, noisier augmentations and that oversampling yields notable gains for Pharo and Python, while quality remains the dominant factor. The work demonstrates a practical, quality-aware data augmentation strategy that improves robustness in code understanding tasks and informs future data-generation practices for developer-oriented NLP problems.

Abstract

Code comments serve a crucial role in software development for documenting functionality, clarifying design choices, and assisting with issue tracking. They capture developers' insights about the surrounding source code, serving as an essential resource for both human comprehension and automated analysis. Nevertheless, since comments are in natural language, they present challenges for machine-based code understanding. To address this, recent studies have applied natural language processing (NLP) and deep learning techniques to classify comments according to developers' intentions. However, existing datasets for this task suffer from size limitations and class imbalance, as they rely on manual annotations and may not accurately represent the distribution of comments in real-world codebases. To overcome this issue, we introduce new synthetic oversampling and augmentation techniques based on high-quality data generation to enhance the NLBSE'26 challenge datasets. Our Synthetic Quality Oversampling Technique and Augmentation Technique (Q-SYNTH) yield promising results, improving the base classifier by $2.56\%$.

High-quality data augmentation for code comment classification

TL;DR

Code comment classification suffers from small, imbalanced datasets across languages in the NLBSE'26 benchmark. The authors propose Q-SYNTH, a three-module framework that generates diverse samples with a BERT-like generator, assesses semantic quality via a sentence-transformer, and selects high-quality samples to balance classes, integrated with a multi-task learning head using the SAMGS optimizer. Their results show that high-quality synthetic data can outperform larger, noisier augmentations and that oversampling yields notable gains for Pharo and Python, while quality remains the dominant factor. The work demonstrates a practical, quality-aware data augmentation strategy that improves robustness in code understanding tasks and informs future data-generation practices for developer-oriented NLP problems.

Abstract

Code comments serve a crucial role in software development for documenting functionality, clarifying design choices, and assisting with issue tracking. They capture developers' insights about the surrounding source code, serving as an essential resource for both human comprehension and automated analysis. Nevertheless, since comments are in natural language, they present challenges for machine-based code understanding. To address this, recent studies have applied natural language processing (NLP) and deep learning techniques to classify comments according to developers' intentions. However, existing datasets for this task suffer from size limitations and class imbalance, as they rely on manual annotations and may not accurately represent the distribution of comments in real-world codebases. To overcome this issue, we introduce new synthetic oversampling and augmentation techniques based on high-quality data generation to enhance the NLBSE'26 challenge datasets. Our Synthetic Quality Oversampling Technique and Augmentation Technique (Q-SYNTH) yield promising results, improving the base classifier by .
Paper Structure (12 sections, 1 equation, 3 figures, 1 table)

This paper contains 12 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Synthetic Quality Technique
  3. Data Generation Module
  4. Quality Scoring Module
  5. Data Selection Module
  6. Computational Experiments
  7. Dataset
  8. Oversampling Specification
  9. Experimental Setting
  10. Results
  11. Final Model Architecture
  12. Conclusion

Figures (3)

  • Figure 1: Comparison of percentage of positive observation across categories in the NLBSE’26 Challenge dataset nlbse2026 and variation produced with the Oversampling technique.
  • Figure 2: Mean F1 score of the baseline model versus the percentage of synthetic data introduced with two alternative strategies (Augmentation or Oversampling). Point size encodes the relative standard deviation. Colors indicate the selected $QSYNT$ level. The red horizontal lines mark the baseline model performance without synthetic data.
  • Figure 3: Distribution of generated sentence token lengths produced by the SetFit tokenizer and their corresponding quality scores ($q$). The central plot shows the joint distribution, while the marginal density plots display the individual distributions of token length and quality score.