Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Business Process Modeling Competences of Large Language Models

Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev

TL;DR

This work introduces BEF4LLM, a BPMN-specific evaluation framework that extends the SIQ quality model with a dedicated validity dimension to assess LLMs generating BPMN from natural language. Through a large-scale benchmark of 17 open-source LLMs on 105 text–BPMN pairs, the study shows that LLMs generally achieve strong syntactic and pragmatic quality, while semantic quality and validity remain challenging, especially for GPT-scale outputs. Larger models improve syntactic and semantic metrics but can degrade pragmatic readability, and instruction-tuned variants frequently outperform non-instruction-tuned ones, guiding practical model selection. Comparisons with human experts reveal that while LLMs can rival humans on several dimensions, semantic validity still benefits from human oversight, underscoring the need for BPMN-specific fine-tuning and reliable validity checks for real-world deployment.

Abstract

The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions. Recent advances in large language models (LLMs) have significantly expanded the possibilities for generating BPMN models directly from natural language, building upon earlier text-to-process methods with enhanced capabilities in handling complex descriptions. However, there is a lack of systematic evaluations of LLM-generated process models. Current efforts either use LLM-as-a-judge approaches or do not consider established dimensions of model quality. To this end, we introduce BEF4LLM, a novel LLM evaluation framework comprising four perspectives: syntactic quality, pragmatic quality, semantic quality, and validity. Using BEF4LLM, we conduct a comprehensive analysis of open-source LLMs and benchmark their performance against human modeling experts. Results indicate that LLMs excel in syntactic and pragmatic quality, while humans outperform in semantic aspects; however, the differences in scores are relatively modest, highlighting LLMs' competitive potential despite challenges in validity and semantic quality. The insights highlight current strengths and limitations of using LLMs for BPMN modeling and guide future model development and fine-tuning. Addressing these areas is essential for advancing the practical deployment of LLMs in business process modeling.

Assessing the Business Process Modeling Competences of Large Language Models

TL;DR

This work introduces BEF4LLM, a BPMN-specific evaluation framework that extends the SIQ quality model with a dedicated validity dimension to assess LLMs generating BPMN from natural language. Through a large-scale benchmark of 17 open-source LLMs on 105 text–BPMN pairs, the study shows that LLMs generally achieve strong syntactic and pragmatic quality, while semantic quality and validity remain challenging, especially for GPT-scale outputs. Larger models improve syntactic and semantic metrics but can degrade pragmatic readability, and instruction-tuned variants frequently outperform non-instruction-tuned ones, guiding practical model selection. Comparisons with human experts reveal that while LLMs can rival humans on several dimensions, semantic validity still benefits from human oversight, underscoring the need for BPMN-specific fine-tuning and reliable validity checks for real-world deployment.

Abstract

The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions. Recent advances in large language models (LLMs) have significantly expanded the possibilities for generating BPMN models directly from natural language, building upon earlier text-to-process methods with enhanced capabilities in handling complex descriptions. However, there is a lack of systematic evaluations of LLM-generated process models. Current efforts either use LLM-as-a-judge approaches or do not consider established dimensions of model quality. To this end, we introduce BEF4LLM, a novel LLM evaluation framework comprising four perspectives: syntactic quality, pragmatic quality, semantic quality, and validity. Using BEF4LLM, we conduct a comprehensive analysis of open-source LLMs and benchmark their performance against human modeling experts. Results indicate that LLMs excel in syntactic and pragmatic quality, while humans outperform in semantic aspects; however, the differences in scores are relatively modest, highlighting LLMs' competitive potential despite challenges in validity and semantic quality. The insights highlight current strengths and limitations of using LLMs for BPMN modeling and guide future model development and fine-tuning. Addressing these areas is essential for advancing the practical deployment of LLMs in business process modeling.
Paper Structure (48 sections, 7 equations, 8 figures, 12 tables)

This paper contains 48 sections, 7 equations, 8 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Business Process Modeling Notation (BPMN)
  4. Foundations and Modeling Elements
  5. Structural Formalization and Notation
  6. Process Connectivity and Control Flow
  7. Process Model Quality
  8. Large Language Models
  9. Related Work
  10. Process Model Quality Frameworks and Assessment
  11. LLMs in BPM
  12. LLM Benchmarks
  13. LLM Benchmarks for BPM Tasks
  14. LLM Benchmarks for Process Modeling
  15. BEF4LLM - BPMN Evaluation Framework for LLMs
  16. ...and 33 more sections

Figures (8)

  • Figure 1: Example of a BPMN Model.
  • Figure 2: SIQ Framework Reijers_Bsuiness_2010.
  • Figure 3: SEQUAL framework Krogstie_sequal_2006.
  • Figure 4: BEF4LLM - BPMN Evaluation Framework for LLMs.
  • Figure 5: Complete Procedure to conducting the experiment for one LLM.
  • ...and 3 more figures