An Empirical Study on LLM-based Classification of Requirements-related Provisions in Food-safety Regulations
Shabnam Hassani, Mehrdad Sabetzadeh, Daniel Amyot
TL;DR
The paper addresses the mismatch between technology-neutral food-safety regulations and the software systems that implement them by first using Grounded Theory to characterize regulation-related concepts relevant to software requirements, and then by building an automated pipeline that classifies provisions with state-of-the-art LLMs. It systematically evaluates BERT and GPT families (notably GPT-4o) against a non-transformer baseline (BiLSTM) and a keyword-based approach, using Canadian SFCR/FSRG data and FDA material to test generalizability. Key findings show that fine-tuned GPT-4o achieves the highest precision and recall (around 89% and 87%, respectively), while GPT-4o with few-shot prompts yields very high recall (about 97%) at the expense of precision (about 65%), highlighting a precision-recall trade-off. The work offers practical value for filtering irrelevant regulatory content and providing metadata-driven guidance for compliance, and it contributes a release-ready dataset and replication package to spur further research in regulatory text analysis for food safety and related domains.
Abstract
As Industry 4.0 transforms the food industry, the role of software in achieving compliance with food-safety regulations is becoming increasingly critical. Food-safety regulations, like those in many legal domains, have largely been articulated in a technology-independent manner to ensure their longevity and broad applicability. However, this approach leaves a gap between the regulations and the modern systems and software increasingly used to implement them. In this article, we pursue two main goals. First, we conduct a Grounded Theory study of food-safety regulations and develop a conceptual characterization of food-safety concepts that closely relate to systems and software requirements. Second, we examine the effectiveness of two families of large language models (LLMs) -- BERT and GPT -- in automatically classifying legal provisions based on requirements-related food-safety concepts. Our results show that: (a) when fine-tuned, the accuracy differences between the best-performing models in the BERT and GPT families are relatively small. Nevertheless, the most powerful model in our experiments, GPT-4o, still achieves the highest accuracy, with an average Precision of 89% and an average Recall of 87%; (b) few-shot learning with GPT-4o increases Recall to 97% but decreases Precision to 65%, suggesting a trade-off between fine-tuning and few-shot learning; (c) despite our training examples being drawn exclusively from Canadian regulations, LLM-based classification performs consistently well on test provisions from the US, indicating a degree of generalizability across regulatory jurisdictions; and (d) for our classification task, LLMs significantly outperform simpler baselines constructed using long short-term memory (LSTM) networks and automatic keyword extraction.