Language Models to Support Multi-Label Classification of Industrial Data

TL;DR

This work tackles the challenge of multi-label requirements classification over large hierarchical taxonomies under data scarcity by evaluating zero-shot classifiers built from a range of language models and language-model hybrids. It introduces a hierarchical embedding approach and a novel label distance metric $D_n$ to assess semantic closeness within the taxonomy, reporting that smaller, SBERT-based autoencoding and sequence-to-sequence models often outperform larger LLMs in embedding quality for semantic similarity. Across 377 requirements and 6 output spaces, the study finds that the best-performing model varies by space, with $T5$-variants excelling in most spaces and $BERT$-base performing best in one, while $D_n$ proves to be a more robust selector than traditional $F_1$ or $F_{eta}$ metrics. The results demonstrate the practical viability of zero-shot, hierarchical multi-label classification in industry, and the proposed distance metric offers a scalable tool for model selection and user-centric taxonomy recommendation, aided by a replication package and data availability statement.

Abstract

Multi-label requirements classification is a challenging task, especially when dealing with numerous classes at varying levels of abstraction. The difficulties increases when a limited number of requirements is available to train a supervised classifier. Zero-shot learning (ZSL) does not require training data and can potentially address this problem. This paper investigates the performance of zero-shot classifiers (ZSCs) on a multi-label industrial dataset. We focuse on classifying requirements according to a taxonomy designed to support requirements tracing. We compare multiple variants of ZSCs using different embeddings, including 9 language models (LMs) with a reduced number of parameters (up to 3B), e.g., BERT, and 5 large LMs (LLMs) with a large number of parameters (up to 70B), e.g., Llama. Our ground truth includes 377 requirements and 1968 labels from 6 output spaces. For the evaluation, we adopt traditional metrics, i.e., precision, recall, F1, and $F_β$, as well as a novel label distance metric Dn. This aims to better capture the classification's hierarchical nature and provides a more nuanced evaluation of how far the results are from the ground truth. 1) The top-performing model on 5 out of 6 output spaces is T5-xl, with maximum $F_β$ = 0.78 and Dn = 0.04, while BERT base outperformed the other models in one case, with maximum $F_β$ = 0.83 and Dn = 0.04. 2) LMs with smaller parameter size produce the best classification results compared to LLMs. Thus, addressing the problem in practice is feasible as limited computing power is needed. 3) The model architecture (autoencoding, autoregression, and sentence-to-sentence) significantly affects the classifier's performance. We conclude that using ZSL for multi-label requirements classification offers promising results. We also present a novel metric that can be used to select the top-performing model for this problem

Figures (3)

• Figure 1: Example of a multi-label requirement classification task with a hierarchical taxonomy.
• Figure 2: Zero-Shot Classifier
• Figure 3: An example demonstrating the measurement of the label distance metric in a taxonomy between p (predicted label) and t (true label)