Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Label Requirements Classification with Large Taxonomies

Waleed Abdeen, Michael Unterkalmsteiner, Krzysztof Wnuk, Alexandros Chirtoglou, Christoph Schimanski, Heja Goli

TL;DR

The paper tackles the challenge of multi-label requirements classification with large taxonomies in infrastructure domains by employing zero-shot learning. It compares a word-based (noun-focused) and a sentence-based (ESA) classifier across six large output spaces, and studies the effects of flat versus hierarchical taxonomy processing. A rigorous ground-truth construction with 129 requirements and 769 labels is built via industry collaboration, enabling evaluation through micro-averaged recall, precision, and F1. Findings show that sentence-based hierarchical classification yields higher recall, but precision remains low, indicating practical limitations; nonetheless, zero-shot approaches offer cost-efficient potential for fine-grained traceability and future improvements using richer preprocessing and larger language models.

Abstract

Classification aids software development activities by organizing requirements in classes for easier access and retrieval. The majority of requirements classification research has, so far, focused on binary or multi-class classification. Multi-label classification with large taxonomies could aid requirements traceability but is prohibitively costly with supervised training. Hence, we investigate zero-short learning to evaluate the feasibility of multi-label requirements classification with large taxonomies. We associated, together with domain experts from the industry, 129 requirements with 769 labels from taxonomies ranging between 250 and 1183 classes. Then, we conducted a controlled experiment to study the impact of the type of classifier, the hierarchy, and the structural characteristics of taxonomies on the classification performance. The results show that: (1) The sentence-based classifier had a significantly higher recall compared to the word-based classifier; however, the precision and F1-score did not improve significantly. (2) The hierarchical classification strategy did not always improve the performance of requirements classification. (3) The total and leaf nodes of the taxonomies have a strong negative correlation with the recall of the hierarchical sentence-based classifier. We investigate the problem of multi-label requirements classification with large taxonomies, illustrate a systematic process to create a ground truth involving industry participants, and provide an analysis of different classification pipelines using zero-shot learning.

Multi-Label Requirements Classification with Large Taxonomies

TL;DR

The paper tackles the challenge of multi-label requirements classification with large taxonomies in infrastructure domains by employing zero-shot learning. It compares a word-based (noun-focused) and a sentence-based (ESA) classifier across six large output spaces, and studies the effects of flat versus hierarchical taxonomy processing. A rigorous ground-truth construction with 129 requirements and 769 labels is built via industry collaboration, enabling evaluation through micro-averaged recall, precision, and F1. Findings show that sentence-based hierarchical classification yields higher recall, but precision remains low, indicating practical limitations; nonetheless, zero-shot approaches offer cost-efficient potential for fine-grained traceability and future improvements using richer preprocessing and larger language models.

Abstract

Classification aids software development activities by organizing requirements in classes for easier access and retrieval. The majority of requirements classification research has, so far, focused on binary or multi-class classification. Multi-label classification with large taxonomies could aid requirements traceability but is prohibitively costly with supervised training. Hence, we investigate zero-short learning to evaluate the feasibility of multi-label requirements classification with large taxonomies. We associated, together with domain experts from the industry, 129 requirements with 769 labels from taxonomies ranging between 250 and 1183 classes. Then, we conducted a controlled experiment to study the impact of the type of classifier, the hierarchy, and the structural characteristics of taxonomies on the classification performance. The results show that: (1) The sentence-based classifier had a significantly higher recall compared to the word-based classifier; however, the precision and F1-score did not improve significantly. (2) The hierarchical classification strategy did not always improve the performance of requirements classification. (3) The total and leaf nodes of the taxonomies have a strong negative correlation with the recall of the hierarchical sentence-based classifier. We investigate the problem of multi-label requirements classification with large taxonomies, illustrate a systematic process to create a ground truth involving industry participants, and provide an analysis of different classification pipelines using zero-shot learning.
Paper Structure (42 sections, 3 equations, 3 figures, 7 tables)

This paper contains 42 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Background
  4. Related work
  5. Multi-label requirements classification with large taxonomies
  6. Word-based classifier (word2vec)
  7. Preprocessing
  8. Exact match predictor
  9. Semantic similarity predictor
  10. Sentence-based classifier (ESA)
  11. Preprocessing
  12. Explicit semantic analysis (ESA)
  13. Relatedness calculation
  14. Global classifier
  15. Adaptations
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Example of a multi-label requirement classification
  • Figure 2: The process of building the ground truth (UML 2.0)
  • Figure 3: Correlation between output space characteristics and the sentence-based hierarchical classifier performance. Each point corresponds to one output space.