Explicating Tacit Regulatory Knowledge from LLMs to Auto-Formalize Requirements for Compliance Test Case Generation

TL;DR

RAFT tackles the manual burden of regulatory compliance testing by explicating tacit regulatory knowledge into explicit artifacts and injecting them into prompts to guide precise requirement formalization and test-case generation. It introduces three artifacts, $\mathcal{M}$, $\mathcal{R}$, and $\mathcal{T}$, and employs Adaptive Purification-Aggregation to fuse multiple LLMs' outputs, enabling reliable, cross-domain test generation. Across finance, automotive, and power, RAFT achieves expert-level effectiveness (≈$F1$ of $91.7\%$ and $BSC$ of $86.5\%$) with substantial efficiency gains (≈$2.3\times$ faster than human experts and up to $134\times$ faster than existing LLM baselines). While showing strong cross-domain generalization, RAFT acknowledges limitations in knowledge-source quality and lacks formal guarantees, suggesting future work in formal verification and richer knowledge bases.

Abstract

Compliance testing in highly regulated domains is crucial but largely manual, requiring domain experts to translate complex regulations into executable test cases. While large language models (LLMs) show promise for automation, their susceptibility to hallucinations limits reliable application. Existing hybrid approaches mitigate this issue by constraining LLMs with formal models, but still rely on costly manual modeling. To solve this problem, this paper proposes RAFT, a framework for requirements auto-formalization and compliance test generation via explicating tacit regulatory knowledge from multiple LLMs. RAFT employs an Adaptive Purification-Aggregation strategy to explicate tacit regulatory knowledge from multiple LLMs and integrate it into three artifacts: a domain meta-model, a formal requirements representation, and testability constraints. These artifacts are then dynamically injected into prompts to guide high-precision requirement formalization and automated test generation. Experiments across financial, automotive, and power domains show that RAFT achieves expert-level performance, substantially outperforms state-of-the-art (SOTA) methods while reducing overall generation and review time.

Figures (8)

• Figure 1: The regulatory knowledge explication and injection-driven framework for fully automated test generation.
• Figure 2: Meta-model for the finance domain.
• Figure 3: An example of 2 regulatory rules from NYSE nyse and their requirement representation.
• Figure 4: Dynamic knowledge injection-based prompt generation for Formal Requirement Generation.
• Figure 5: Comparison of the time consumption on knowledge explication (Exp.), test generation (Gen.), and expert review (Rev.) on processing the six datasets.