Automatic techniques for issue report classification: A systematic mapping study

TL;DR

The paper surveys automatic techniques for classifying issue reports through a systematic mapping of 46 primary studies. It reveals a spectrum of traditional ML, deep learning, and large language model approaches, with RoBERTa/BERT-family models often achieving top performance, yet many studies rely solely on prediction accuracy and OSS data. It highlights gaps in industrial validation and practitioner involvement and calls for research that incorporates adoption factors such as explainability, scalability, and generalizability, as well as robust evaluation protocols. The work provides a detailed literature map and practical guidance for researchers and practitioners about when and how to apply automatic issue report classification in real-world settings.

Abstract

Several studies have evaluated automatic techniques for classifying software issue reports to assist practitioners in effectively assigning relevant resources based on the type of issue. Currently, no comprehensive overview of this area has been published. A comprehensive overview will help identify future research directions and provide an extensive collection of potentially relevant existing solutions. This study aims to provide a comprehensive overview of the use of automatic techniques to classify issue reports. We conducted a systematic mapping study and identified 46 studies on the topic. The study results indicate that the existing literature applies various techniques for classifying issue reports, including traditional machine learning and deep learning-based techniques and more advanced large language models. Furthermore, we observe that these studies (a) lack the involvement of practitioners, (b) do not consider other potentially relevant adoption factors beyond prediction accuracy, such as the explainability, scalability, and generalizability of the techniques, and (c) mainly rely on archival data from open-source repositories only. Therefore, future research should focus on real industrial evaluations, consider other potentially relevant adoption factors, and actively involve practitioners.

Figures (3)

• Figure 1: Overview of our approach
• Figure 2: Yearly distribution of primary studies grouped by their publication venue type
• Figure 3: Categorization of automatic techniques for issue report classification