MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training

TL;DR

Noise in training data undermines code-model robustness in SE tasks. MANTRA introduces a two-stage framework that first analyzes noise-induced training dynamics and then applies Gaussian-mixture filtering with adaptive dropout to drop noisy samples during fine-tuning. Across code summarization and commit-intent classification with five diverse models, MANTRA reduces loss-trajectory disruption and mitigates performance degradation under label noise. The work provides replication materials and demonstrates practical benefits for data-quality management in software engineering.

Abstract

The reliable application of deep learning models to software engineering tasks hinges on high-quality training data. Yet, large-scale repositories inevitably introduce noisy or mislabeled examples that degrade both accuracy and robustness. While Noise Label Learning (NLL) has been extensively studied in other fields, there are a few works that investigate NLL in Software Engineering (SE) and Large Language Models (LLMs) for SE tasks. In this work, we propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process of code-Pretrained Language Models (PTM) and code-LLMs. We first investigate the effect of noise at varying levels on convergence and loss trajectories of the models. Then we apply an adaptive dropout strategy guided by per-sample loss dynamics and Gaussian Mixture Model clustering to exclude persistently noisy points while preserving clean data. Applying to code summarization and commit intent classification, our experiments reveal that some LLMs are more sensitive to noise than others. However, with MANTRA, the performance of all models in both tasks is improved. MANTRA enables researchers and practitioners to reduce the impact of errors introduced by the dataset in training, saves time in data cleaning and processing, while maximizing the effect of fine-tuning.

Figures (9)

• Figure 1: The workflow for this study. P1 explores loss distribution and noise behaviour, P2 applies MANTRA during training.
• Figure 2: Distribution of loss for CodeBERT and CodeT5+ at 5% across epochs 1, 2, 3, and 10.
• Figure 3: Loss density distributions by model, epoch, and noise level for code summarization. Each subplot has independent axes.
• Figure 4: Loss density distributions by model, epoch, and noise level for code commit intent classification. Each subplot has independent axes.
• Figure 5: Loss density distributions for CodeT5+ at epochs 7 and 9 under 15% noise insertion on a log scale.