On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study
Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto
TL;DR
The paper tackles whether code comments enhance ABF when using LLMs by building a renovated, comment-preserving dataset and generating high-quality comments for commented-poor instances. It evaluates two ABF-focused model families (CodeT5+ and DeepSeek-Coder) across all combinations of training-time and inference-time comments, finding that comments boost ABF accuracy significantly when present in both stages, with notable gains even when only present at inference. Training with comments does not harm performance on uncommented instances, and the strongest improvements occur when comments are available in both phases. An interpretability analysis using SHAP reveals that comments describing how the method achieves its goal are particularly influential, underscoring that code documentation can meaningfully aid LLM-assisted bug fixing and should be encouraged in practice.
Abstract
Large Language Models (LLMs) are increasingly relevant in Software Engineering research and practice, with Automated Bug Fixing (ABF) being one of their key applications. ABF involves transforming a buggy method into its fixed equivalent. A common preprocessing step in ABF involves removing comments from code prior to training. However, we hypothesize that comments may play a critical role in fixing certain types of bugs by providing valuable design and implementation insights. In this study, we investigate how the presence or absence of comments, both during training and at inference time, impacts the bug-fixing capabilities of LLMs. We conduct an empirical evaluation comparing two model families, each evaluated under all combinations of training and inference conditions (with and without comments), and thereby revisiting the common practice of removing comments during training. To address the limited availability of comments in state-of-the-art datasets, we use an LLM to automatically generate comments for methods lacking them. Our findings show that comments improve ABF accuracy by up to threefold when present in both phases, while training with comments does not degrade performance when instances lack them. Additionally, an interpretability analysis identifies that comments detailing method implementation are particularly effective in aiding LLMs to fix bugs accurately.