LLM4PM: A case study on using Large Language Models for Process Modeling in Enterprise Organizations

TL;DR

Process modeling in BPM is costly due to fragmented documentation. The authors develop PRODIGY, an LLM-based chatbot guided by Design Science Research to assist Hilti modellers by leveraging internal documents through retrieval-augmented generation and BPMN Sketch Miner compatibility. They conduct a Hilti-focused user study, propose an operating governance model, and provide practical lessons for deploying enterprise LLMs in BPM. The work demonstrates both the promise and the challenges of context-specific, governance-enabled LLM deployments for business process management.

Abstract

We investigate the potential of using Large Language Models (LLM) to support process model creation in organizational contexts. Specifically, we carry out a case study wherein we develop and test an LLM-based chatbot, PRODIGY (PROcess moDellIng Guidance for You), in a multinational company, the Hilti Group. We are particularly interested in understanding how LLM can aid (human) modellers in creating process flow diagrams. To this purpose, we first conduct a preliminary user study (n=10) with professional process modellers from Hilti, inquiring for various pain-points they encounter in their daily routines. Then, we use their responses to design and implement PRODIGY. Finally, we evaluate PRODIGY by letting our user study's participants use PRODIGY, and then ask for their opinion on the pros and cons of PRODIGY. We coalesce our results in actionable takeaways. Through our research, we showcase the first practical application of LLM for process modelling in the real world, shedding light on how industries can leverage LLM to enhance their Business Process Management activities.

Figures (6)

• Figure 1: An exemplary usage of PRODIGY -- Our LLM-powered chatbot can fulfill various BPM-related tasks. Its most appreciated functionality is generating an output for a complete process model (if copy-pasted into the open-source tool BPMN Sketch Miner ivanchikj2022live)
• Figure 2: Method. We rely on design science research to design, develop, and deploy our artifact. During the implementation of PRODIGY, we also devise an "operating model" (which we validate in the evaluation of PRODIGY) through which we explain how PRODIGY should be used in real organizations.
• Figure 3: Helpfulness of PRODIGY's functionalities. According to our participants, most of our implemented features are helpful---especially for creating process models and supporting human requests.
• Figure 4: Operating model of PRODIGY -- We visualize the interactions between the Governance Team (e.g., developers and managers) of a given organization with the Process Modeller (i.e., the end-users of PRODIGY) that ensure a smooth operation of PRODIGY for real-world deployments.
• Figure 5: Issues of process modellers. Our framework has three levels with two-way transitions between each level---each having its own set of issues. Model value$\rightarrow$Model creation: the organization must communicate a clear strategic directive for process model creation, balancing the cost/benefit of model creation and usage [issue: during the "Prep phase", process modellers find existing communication to lack clarity, leading to time waste and unproductive discussions among various stakeholders]. Model creation$\rightarrow$Model value: While creating the model, process modelers should have a clear vision of "who and how" is going to use the model (which is what leads to the model becoming valuable) [issue: lack of clarity and/or poor documentation may lead to process models representing "standalone exercises" which do not bring any value to the company.]