Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis

TL;DR

This work investigates the use of large language models (LLMs) for automated business process modeling within a structured framework. It introduces ProMoAI, a framework that uses an intermediate POWL representation, a 20‑process ground-truth benchmark, and a simulation-based conformance evaluation to compare 16 LLMs across tasks of natural-language understanding, code generation, and iterative refinement. It further analyzes LLM self-improvement strategies—self-evaluation, input optimization, and output optimization—finding that output optimization offers the strongest gains, especially for lower‑performing models, while self-evaluation and input optimization show mixed results. Key findings reveal substantial performance variation across LLM families, and a positive link between efficient error handling and higher quality conformance, guiding practical choices for BPM applications. The results provide a foundation for more automated, reliable BPM with LLMs and point to directions for expanding perspectives, direct BPMN generation, and richer prompting and knowledge sources.

Abstract

Large Language Models (LLMs) are rapidly transforming various fields, and their potential in Business Process Management (BPM) is substantial. This paper assesses the capabilities of LLMs on business process modeling using a framework for automating this task, a comprehensive benchmark, and an analysis of LLM self-improvement strategies. We present a comprehensive evaluation of 16 state-of-the-art LLMs from major AI vendors using a custom-designed benchmark of 20 diverse business processes. Our analysis highlights significant performance variations across LLMs and reveals a positive correlation between efficient error handling and the quality of generated models. It also shows consistent performance trends within similar LLM groups. Furthermore, we investigate LLM self-improvement techniques, encompassing self-evaluation, input optimization, and output optimization. Our findings indicate that output optimization, in particular, offers promising potential for enhancing quality, especially in models with initially lower performance. Our contributions provide insights for leveraging LLMs in BPM, paving the way for more advanced and automated process modeling techniques.

Figures (6)

• Figure 1: LLM-based process modeling framework.
• Figure 2: A screenshot of ProMoAI.
• Figure 3: Ground truth and LLM-generated process models for process p9.
• Figure 4: LLM-generated process models for process p9.
• Figure 5: Ground truth and LLM-generated process models for process p18.