From Law to Gherkin: A Human-Centred Quasi-Experiment on the Quality of LLM-Generated Behavioural Specifications from Food-Safety Regulations

TL;DR

This is the first systematic human-subject evaluation of LLMs'ability to derive Gherkin behavioural specifications from legal texts using a quasi-experimental design and noted occasional omissions, hallucinations, and mixed intents, underscoring the need for human oversight, especially in safety-critical domains.

Abstract

Context: Laws and regulations increasingly shape software design, development, and quality assurance in regulated domains. Because legal provisions are written in technology-neutral language, deriving concrete specifications, requirements, and acceptance criteria to verify software compliance is difficult and error-prone. Recent advances in generative AI, especially large language models (LLMs), may help automate this process. Objective: We present the first systematic human-subject evaluation of LLMs' ability to derive Gherkin behavioural specifications from legal texts using a quasi-experimental design. Gherkin is a domain-specific language for scenario-based system behaviour descriptions in Given-When-Then form and is well suited to automation in software development. Methods: Ten participants evaluated 60 Gherkin specifications generated from food-safety regulations by Claude and Llama. Each participant assessed 12 specifications across five criteria: relevance, clarity, completeness, singularity, and time savings. Each specification was evaluated by two participants, yielding 120 assessments with quantitative ratings and qualitative feedback. Results: Ratings were uniformly high in the top two categories: relevance 95%, clarity 100%, completeness 94.2%, singularity 93.4%, and time savings 91.7%. No statistically reliable differences were found across participants or between LLMs. Qualitative feedback noted occasional omissions, hallucinations, and mixed intents, underscoring the need for human oversight, especially in safety-critical domains. Conclusion: In food safety, LLMs can assist in deriving Gherkin specifications from legal texts, but omissions and hallucinations require systematic human review.

Figures (10)

• Figure 1: Example Gherkin specifications generated by two different LLMs for the same food-safety legal provision: (a) illustrates singular and focused scenarios; (b) illustrates mixed-objective scenarios.
• Figure 2: Prompt for generating a Gherkin specification.
• Figure 3: Scatter plot of legal-provision token counts versus corresponding Gherkin specification token counts for Llama and Claude, with fitted regression lines.
• Figure 5: Stacked bar plots per criterion for all evaluations across both LLMs combined and stacked bar plots per criterion per LLM.
• Figure 6: Issue themes observed in LLM-generated Gherkin specifications, grouped by quality criteria. Numbers in parentheses indicate the counts of participant comments. An individual comment could relate to multiple quality criteria if it addressed more than one type of issue simultaneously.