Identifying Technical Debt and Its Types Across Diverse Software Projects Issues

TL;DR

The study tackles automated TD detection and TD-type classification from issue trackers using transformer-based models, comparing DistilRoBERTa and GPTs, and evaluating binary versus multiclass approaches with extensive open-source and industrial data. It demonstrates that project-specific fine-tuning substantially boosts performance, and that specialized binary classifiers for TD types generally outperform a single multiclass model. DistilRoBERTa frequently surpasses GPT-based models in TD tasks while offering better efficiency, and industrial validation on Visma data confirms practical applicability. The work also emphasizes generalization to out-of-distribution data and provides a public dataset release to advance TD research and tooling.

Abstract

Technical Debt (TD) identification in software projects issues is crucial for maintaining code quality, reducing long-term maintenance costs, and improving overall project health. This study advances TD classification using transformer-based models, addressing the critical need for accurate and efficient TD identification in large-scale software development. Our methodology employs multiple binary classifiers for TD and its type, combined through ensemble learning, to enhance accuracy and robustness in detecting various forms of TD. We train and evaluate these models on a comprehensive dataset from GitHub Archive Issues (2015-2024), supplemented with industrial data validation. We demonstrate that in-project fine-tuned transformer models significantly outperform task-specific fine-tuned models in TD classification, highlighting the importance of project-specific context in accurate TD identification. Our research also reveals the superiority of specialized binary classifiers over multi-class models for TD and its type identification, enabling more targeted debt resolution strategies. A comparative analysis shows that the smaller DistilRoBERTa model is more effective than larger language models like GPTs for TD classification tasks, especially after fine-tuning, offering insights into efficient model selection for specific TD detection tasks. The study also assesses generalization capabilities using metrics such as MCC, AUC ROC, Recall, and F1 score, focusing on model effectiveness, fine-tuning impact, and relative performance. By validating our approach on out-of-distribution and real-world industrial datasets, we ensure practical applicability, addressing the diverse nature of software projects.