Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Applying Large Language Models API to Issue Classification Problem

Gabriel Aracena, Kyle Luster, Fabio Santos, Igor Steinmacher, Marco A. Gerosa

TL;DR

The paper tackles automatic prioritization of OSS issue reports by classifying issue types (bug, feature, question) using GPT-based fine-tuning with limited training data. It builds a pipeline that includes careful data preprocessing, per-repo fine-tuning of gpt-3.5-turbo via the OpenAI API, and evaluation using precision, recall, and F1. Results show a macro-average precision of 83.24%, recall of 82.87%, and F1 of 82.80% when using issue titles and bodies, with per-repo F1 ranging from 76.65% to 87.08%. Compared to a baseline, the approach is competitive, though performance varies with data quality and labeling clarity, underscoring the importance of standardized labeling for practical deployment.

Abstract

Effective prioritization of issue reports is crucial in software engineering to optimize resource allocation and address critical problems promptly. However, the manual classification of issue reports for prioritization is laborious and lacks scalability. Alternatively, many open source software (OSS) projects employ automated processes for this task, albeit relying on substantial datasets for adequate training. This research seeks to devise an automated approach that ensures reliability in issue prioritization, even when trained on smaller datasets. Our proposed methodology harnesses the power of Generative Pre-trained Transformers (GPT), recognizing their potential to efficiently handle this task. By leveraging the capabilities of such models, we aim to develop a robust system for prioritizing issue reports accurately, mitigating the necessity for extensive training data while maintaining reliability. In our research, we have developed a reliable GPT-based approach to accurately label and prioritize issue reports with a reduced training dataset. By reducing reliance on massive data requirements and focusing on few-shot fine-tuning, our methodology offers a more accessible and efficient solution for issue prioritization in software engineering. Our model predicted issue types in individual projects up to 93.2% in precision, 95% in recall, and 89.3% in F1-score.

Applying Large Language Models API to Issue Classification Problem

TL;DR

The paper tackles automatic prioritization of OSS issue reports by classifying issue types (bug, feature, question) using GPT-based fine-tuning with limited training data. It builds a pipeline that includes careful data preprocessing, per-repo fine-tuning of gpt-3.5-turbo via the OpenAI API, and evaluation using precision, recall, and F1. Results show a macro-average precision of 83.24%, recall of 82.87%, and F1 of 82.80% when using issue titles and bodies, with per-repo F1 ranging from 76.65% to 87.08%. Compared to a baseline, the approach is competitive, though performance varies with data quality and labeling clarity, underscoring the importance of standardized labeling for practical deployment.

Abstract

Effective prioritization of issue reports is crucial in software engineering to optimize resource allocation and address critical problems promptly. However, the manual classification of issue reports for prioritization is laborious and lacks scalability. Alternatively, many open source software (OSS) projects employ automated processes for this task, albeit relying on substantial datasets for adequate training. This research seeks to devise an automated approach that ensures reliability in issue prioritization, even when trained on smaller datasets. Our proposed methodology harnesses the power of Generative Pre-trained Transformers (GPT), recognizing their potential to efficiently handle this task. By leveraging the capabilities of such models, we aim to develop a robust system for prioritizing issue reports accurately, mitigating the necessity for extensive training data while maintaining reliability. In our research, we have developed a reliable GPT-based approach to accurately label and prioritize issue reports with a reduced training dataset. By reducing reliance on massive data requirements and focusing on few-shot fine-tuning, our methodology offers a more accessible and efficient solution for issue prioritization in software engineering. Our model predicted issue types in individual projects up to 93.2% in precision, 95% in recall, and 89.3% in F1-score.
Paper Structure (17 sections, 1 figure, 2 tables)

This paper contains 17 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Data Preprocessing and Cleaning
  4. Introduction to the Dataset
  5. Attributes of Each Issue Report
  6. Labeling and Exclusions
  7. Noise Removal and Data Preparation
  8. Data Segmentation and Labeling
  9. Format Conversion for Model Input
  10. Model Implementation and Training
  11. Invoking the API
  12. Model Fine-tuning Process
  13. Performance Evaluation Metrics
  14. Getting the results
  15. Results
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Method