Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Effectiveness of LLMs in Automated Self-Admitted Technical Debt Repayment

Mohammad Sadegh Sheikhaei, Yuan Tian, Shaowei Wang, Bowen Xu

TL;DR

This work benchmarks automated SATD repayment by constructing large, language-independent datasets for Python and Java using SATD Tracker, and by introducing diff-based evaluation metrics (BLEU-diff, CrystalBLEU-diff) plus Line-Level Exact Match on Diff (LEMOD). It demonstrates that dataset cleanliness and robust evaluation metrics reveal stronger promise for LLM-based SATD repayment, with prompt-based large models often outperforming fine-tuned small models in EM and other metrics. The study also shows that an oracle, or dynamically chosen prompt template, can significantly boost performance, and that EM alone may underrepresent model capability, motivating the use of LEMOD alongside EM. Overall, the paper provides new benchmarks, metrics, and comparative insights that advance automated SATD repayment research and point to future directions in adaptive prompting and multilingual support.

Abstract

Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality and increase maintenance costs. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored. In this paper, we identify three key challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayments, and (3) limitations of existing evaluation metrics. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repositories, resulting two large-scale datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code. Additionally, we propose another new metric, LEMOD, which is both interpretable and informative. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models, and prompt engineering with five large-scale models. Our results reveal that fine-tuned small models achieve comparable Exact Match (EM) scores to prompt-based approaches but underperform on BLEU-based metrics and LEMOD. Notably, Gemma-2-9B leads in EM, addressing 10.1% of Python and 8.1% of Java SATDs, while Llama-3.1-70B-Instruct and GPT-4o-mini excel on BLEU-diff, CrystalBLEU-diff, and LEMOD metrics. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.

Understanding the Effectiveness of LLMs in Automated Self-Admitted Technical Debt Repayment

TL;DR

This work benchmarks automated SATD repayment by constructing large, language-independent datasets for Python and Java using SATD Tracker, and by introducing diff-based evaluation metrics (BLEU-diff, CrystalBLEU-diff) plus Line-Level Exact Match on Diff (LEMOD). It demonstrates that dataset cleanliness and robust evaluation metrics reveal stronger promise for LLM-based SATD repayment, with prompt-based large models often outperforming fine-tuned small models in EM and other metrics. The study also shows that an oracle, or dynamically chosen prompt template, can significantly boost performance, and that EM alone may underrepresent model capability, motivating the use of LEMOD alongside EM. Overall, the paper provides new benchmarks, metrics, and comparative insights that advance automated SATD repayment research and point to future directions in adaptive prompting and multilingual support.

Abstract

Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality and increase maintenance costs. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored. In this paper, we identify three key challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayments, and (3) limitations of existing evaluation metrics. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repositories, resulting two large-scale datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code. Additionally, we propose another new metric, LEMOD, which is both interpretable and informative. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models, and prompt engineering with five large-scale models. Our results reveal that fine-tuned small models achieve comparable Exact Match (EM) scores to prompt-based approaches but underperform on BLEU-based metrics and LEMOD. Notably, Gemma-2-9B leads in EM, addressing 10.1% of Python and 8.1% of Java SATDs, while Llama-3.1-70B-Instruct and GPT-4o-mini excel on BLEU-diff, CrystalBLEU-diff, and LEMOD metrics. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.
Paper Structure (26 sections, 2 equations, 2 figures, 18 tables)

This paper contains 26 sections, 2 equations, 2 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Empirical Study on SATD Removal and Automated SATD Repayment
  4. Other Studies on SATD Management
  5. Benchmarking LLMs on SE Tasks
  6. Empirical Study Design
  7. RQ1- Can we build a benchmark for automated SATD repayment that is more representative and less noisy?
  8. RQ2- Can we design evaluation metrics that are both effective and interpretable for automated SATD repayment?
  9. RQ3- Does a cleaner dataset lead to a better performance in automated SATD repayment?
  10. RQ4- How do fine-tuned small models compare to prompt-based approaches using large LLMs for automated SATD repayment?
  11. Study Setup
  12. Datasets
  13. Evaluation Metrics
  14. Selected Large Language Models
  15. Implementation
  16. ...and 11 more sections

Figures (2)

  • Figure 2: Prompt used in final filtering step to detect relevance of code updates to SATD repayment
  • Figure 3: Comparison of BLEU and CrystalBLEU scores on the whole code versus on the diffs.