Self-Admitted Technical Debt in LLM Software: An Empirical Comparison with ML and Non-ML Software

TL;DR

This study presents the first empirical analysis of self-admitted technical debt (SATD) in LLM-based software and compares it with ML and traditional software. By assembling 159 post-ChatGPT LLM repositories alongside matched ML and non-ML cohorts and analyzing over 5 million comment events, the authors show that SATD prevalence in LLM systems is similar to ML, but LLM projects enjoy a longer debt-free period before debt accumulates and persist longer once introduced. They introduce three LLM-specific debt categories—Model-Stack Workaround Debt, Model Dependency Debt, and Performance Optimization Debt—and map debt concentration to Deployment/Monitoring and Pretraining stages. The work provides a reproducible replication package, extends SATD taxonomy for modern AI systems, and offers practical guidance for prioritizing debt remediation in LLM pipelines, informing both researchers and practitioners about paradigm-specific debt dynamics and management strategies.

Abstract

Self-admitted technical debt (SATD), referring to comments flagged by developers that explicitly acknowledge suboptimal code or incomplete functionality, has received extensive attention in machine learning (ML) and traditional (Non-ML) software. However, little is known about how SATD manifests and evolves in contemporary Large Language Model (LLM)-based systems, whose architectures, workflows, and dependencies differ fundamentally from both traditional and pre-LLM ML software. In this paper, we conduct the first empirical study of SATD in the LLM era, replicating and extending prior work on ML technical debt to modern LLM-based systems. We compare SATD prevalence across LLM, ML, and non-ML repositories across a total of 477 repositories (159 per category). We perform survival analysis of SATD introduction and removal to understand the dynamics of technical debt across different development paradigms. Surprisingly, despite their architectural complexity, our results reveal that LLM repositories accumulate SATD at similar rates to ML systems (3.95% vs. 4.10%). However, we observe that LLM repositories remain debt-free 2.4x longer than ML repositories (a median of 492 days vs. 204 days), and then start to accumulate technical debt rapidly. Moreover, our qualitative analysis of 377 SATD instances reveals three new forms of technical debt unique to LLM-based development that have not been reported in prior research: Model-Stack Workaround Debt, Model Dependency Debt, and Performance Optimization Debt. Finally, by mapping SATD to stages of the LLM development pipeline, we observe that debt concentrates

Figures (3)

• Figure 1: Distribution of SATD types in LLM repositories.
• Figure 2: Distribution of SATD Comments Across LLM Development Pipeline Stages
• Figure 3: Kaplan–Meier survival curves for SATD introduction (left) and removal (right).