Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

TL;DR

This work tackles rule-flow modeling in business processes by introducing the BPRF dataset and the ExIde framework. BPRF provides 50 Chinese documents with 326 explicitly labeled <Condition, Action> rules and dependency annotations (sequential, conditional, parallel), enabling robust evaluation of rule extraction and dependency classification. ExIde employs a two-stage, prompt-engineered LLM pipeline to (i) extract discrete rules and (ii) infer inter-rule dependencies across 12 SOTA LLMs, showing that carefully designed prompts and reasoning-oriented models improve performance. The contributions advance automated, interpretable business process automation by bridging rule-centric logic with large language models, while acknowledging limitations in scale and prompt sensitivity that warrant future expansion and integration with downstream decision-support systems.

Abstract

Process mining aims to discover, monitor and optimize the actual behaviors of real processes. While prior work has mainly focused on extracting procedural action flows from instructional texts, rule flows embedded in business documents remain underexplored. To this end, we introduce a novel annotated Chinese dataset, BPRF, which contains 50 business process documents with 326 explicitly labeled business rules across multiple domains. Each rule is represented as a <Condition, Action> pair, and we annotate logical dependencies between rules (sequential, conditional, or parallel). We also propose ExIde, a framework for automatic business rule extraction and dependency relationship identification using large language models (LLMs). We evaluate ExIde using 12 state-of-the-art (SOTA) LLMs on the BPRF dataset, benchmarking performance on both rule extraction and dependency classification tasks of current LLMs. Our results demonstrate the effectiveness of ExIde in extracting structured business rules and analyzing their interdependencies for current SOTA LLMs, paving the way for more automated and interpretable business process automation.

Figures (15)

• Figure 1: Annotation process of the BPRF dataset, which includes 1). Data Collection combining collections from the internet and synthetic generation using ChatGPT; 2). Annotation with three domain experts; 3). Verification with another three experts and 4). Evaluation for inter-annotator agreement using ICC and IAA.
• Figure 2: The human evaluation results from every perspective among three annotators.
• Figure 3: Framework overview of ExIde, which includes two stages: 1). 5 different prompt variants are designed to guide LLMs to extract business rules and 2). a dedicated prompt is designed for dependency relationship identification among the extracted business rules from the stage 1.
• Figure 4: Illustration of a sequential dependency in a banking service scenario. The process follows a strict linear order: currency type $\rightarrow$ transaction type (cash or remittance) $\rightarrow$ purchase amount. Each step is dependent on the completion of the previous one, forming a chain-like execution flow.
• Figure 5: Illustration of a conditional dependency in a car rental scenario. The flow diverges based on the vehicle type selected. A sedan or SUV leads to rental duration selection, while a business vehicle or RV leads to rental purpose selection. Downstream steps, such as insurance purchase, follow the resolution of these conditional branches.