Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Far Have LLMs Come Toward Automated SATD Taxonomy Construction?

Sota Nakashima, Yuta Ishimoto, Masanari Kondo, Tao Xiao, Yasutaka Kamei

TL;DR

This work investigates using large language models to automate SATD taxonomy construction by detailing a two-phase pipeline that first explains SATD comments in code context and then builds a two-level taxonomy through generation and update steps. It evaluates the approach on three domain datasets (quantum software, smart contracts, and ML software) and shows that domain-specific categories can be recovered, while also outperforming a naive LLM baseline in several metrics. The method completes taxonomy generation for 448 comments in under two hours at a cost of less than $1, highlighting substantial efficiency gains over manual approaches. The findings suggest practical semi-automated workflows for taxonomy construction and point to future human-in-the-loop extensions and cross-domain applicability.

Abstract

Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. In this study, we investigated to what extent large language models (LLMs) could generate SATD taxonomies. We designed a structured, LLM-driven pipeline that mirrors the taxonomy construction steps researchers typically follow. We evaluated it on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovered domain-specific categories reported in prior work, such as Layer Configuration in machine learning. It also completed taxonomy generation in under two hours and for less than $1, even on the largest dataset. These results suggest that, while full automation remains challenging, LLMs can support semi-automated SATD taxonomy construction. Furthermore, our work opens up avenues for future work, such as automated taxonomy generation in other areas.

How Far Have LLMs Come Toward Automated SATD Taxonomy Construction?

TL;DR

This work investigates using large language models to automate SATD taxonomy construction by detailing a two-phase pipeline that first explains SATD comments in code context and then builds a two-level taxonomy through generation and update steps. It evaluates the approach on three domain datasets (quantum software, smart contracts, and ML software) and shows that domain-specific categories can be recovered, while also outperforming a naive LLM baseline in several metrics. The method completes taxonomy generation for 448 comments in under two hours at a cost of less than $1, highlighting substantial efficiency gains over manual approaches. The findings suggest practical semi-automated workflows for taxonomy construction and point to future human-in-the-loop extensions and cross-domain applicability.

Abstract

Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. In this study, we investigated to what extent large language models (LLMs) could generate SATD taxonomies. We designed a structured, LLM-driven pipeline that mirrors the taxonomy construction steps researchers typically follow. We evaluated it on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovered domain-specific categories reported in prior work, such as Layer Configuration in machine learning. It also completed taxonomy generation in under two hours and for less than $1, even on the largest dataset. These results suggest that, while full automation remains challenging, LLMs can support semi-automated SATD taxonomy construction. Furthermore, our work opens up avenues for future work, such as automated taxonomy generation in other areas.

Paper Structure

This paper contains 26 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SATD Taxonomy and Automation
  4. Taxonomy Construction.
  5. LLM-driven pipeline for SATD taxonomy generation
  6. Overview
  7. Phase 1: Generating Explanations for SATD
  8. Phase 2: Generating/Updating Categories at Each Level
  9. Generation Step
  10. Update Step
  11. Study Design
  12. Research Questions
  13. Studied Datasets
  14. Selected Models
  15. Prompt Design
  16. ...and 11 more sections

Figures (4)

  • Figure 1: SATD detection, classification, and taxonomy generation.
  • Figure 2: The LLM-driven pipeline for SATD taxonomy generation. The robot icons above the arrows indicate the steps in which LLMs are utilized.
  • Figure 3: Example Sankey diagram of human-defined and LLM-generated categories (Dataset: QS, Level: Main, Run ID: 0).
  • Figure 4: Distribution of $top\_sim(H_i)$ scores across datasets and taxonomy levels. The annotated examples (e.g., Algorithm, Layer Configuration) correspond to the top-5 highest similarity scores across all data points.