Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Universal Prompting Strategy for Extracting Process Model Information from Natural Language Text using Large Language Models

Julian Neuberger, Lars Ackermann, Han van der Aa, Stefan Jablonski

TL;DR

The paper tackles the challenge of extracting BPM-relevant information from natural language when data for traditional learning approaches is scarce. It introduces a novel, modular prompting strategy for large language models that combines Context, Task Description, and Restrictions, and demonstrates a proof-of-concept pipeline to transform extracted data into BPMN or declarative process models. Across three datasets and eight LLMs, the approach achieves up to $8\%$ absolute improvements in $F_1$ over state-of-the-art baselines and shows strong cross-model generality, including zero-shot performance that rivals supervised methods. The work provides practical guidelines for prompt design and releases code, prompts, and data to support further research and real-world adoption.

Abstract

Over the past decade, extensive research efforts have been dedicated to the extraction of information from textual process descriptions. Despite the remarkable progress witnessed in natural language processing (NLP), information extraction within the Business Process Management domain remains predominantly reliant on rule-based systems and machine learning methodologies. Data scarcity has so far prevented the successful application of deep learning techniques. However, the rapid progress in generative large language models (LLMs) makes it possible to solve many NLP tasks with very high quality without the need for extensive data. Therefore, we systematically investigate the potential of LLMs for extracting information from textual process descriptions, targeting the detection of process elements such as activities and actors, and relations between them. Using a heuristic algorithm, we demonstrate the suitability of the extracted information for process model generation. Based on a novel prompting strategy, we show that LLMs are able to outperform state-of-the-art machine learning approaches with absolute performance improvements of up to 8\% $F_1$ score across three different datasets. We evaluate our prompting strategy on eight different LLMs, showing it is universally applicable, while also analyzing the impact of certain prompt parts on extraction quality. The number of example texts, the specificity of definitions, and the rigour of format instructions are identified as key for improving the accuracy of extracted information. Our code, prompts, and data are publicly available.

A Universal Prompting Strategy for Extracting Process Model Information from Natural Language Text using Large Language Models

TL;DR

The paper tackles the challenge of extracting BPM-relevant information from natural language when data for traditional learning approaches is scarce. It introduces a novel, modular prompting strategy for large language models that combines Context, Task Description, and Restrictions, and demonstrates a proof-of-concept pipeline to transform extracted data into BPMN or declarative process models. Across three datasets and eight LLMs, the approach achieves up to absolute improvements in over state-of-the-art baselines and shows strong cross-model generality, including zero-shot performance that rivals supervised methods. The work provides practical guidelines for prompt design and releases code, prompts, and data to support further research and real-world adoption.

Abstract

Over the past decade, extensive research efforts have been dedicated to the extraction of information from textual process descriptions. Despite the remarkable progress witnessed in natural language processing (NLP), information extraction within the Business Process Management domain remains predominantly reliant on rule-based systems and machine learning methodologies. Data scarcity has so far prevented the successful application of deep learning techniques. However, the rapid progress in generative large language models (LLMs) makes it possible to solve many NLP tasks with very high quality without the need for extensive data. Therefore, we systematically investigate the potential of LLMs for extracting information from textual process descriptions, targeting the detection of process elements such as activities and actors, and relations between them. Using a heuristic algorithm, we demonstrate the suitability of the extracted information for process model generation. Based on a novel prompting strategy, we show that LLMs are able to outperform state-of-the-art machine learning approaches with absolute performance improvements of up to 8\% score across three different datasets. We evaluate our prompting strategy on eight different LLMs, showing it is universally applicable, while also analyzing the impact of certain prompt parts on extraction quality. The number of example texts, the specificity of definitions, and the rigour of format instructions are identified as key for improving the accuracy of extracted information. Our code, prompts, and data are publicly available.
Paper Structure (18 sections, 4 figures, 3 tables)

This paper contains 18 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Task Descriptions and Challenges
  3. Task descriptions
  4. Challenges of Process Information Extraction from Text
  5. Challenges of Process Information Extraction Using LLMs
  6. Related Work
  7. LLM-based Process Model Extraction from Text
  8. Process Information Extraction with LLMs
  9. High-Level Prompt Structure.
  10. Context and Task Description.
  11. Restrictions.
  12. Process Model Generation - a Proof of Concept
  13. Experiment Setup
  14. Results
  15. Model Comparison
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Fragment of a larger text describing a business process of an insurance company. Different methodologies may extract different process relevant information, depending on the target modelling notation or use case.
  • Figure 2: Modular prompt structure (underlined = task-specific content, boxes = design pattern, = useful, = non-useful, = use in prompt engineering).
  • Figure 3: Automatically generated model for information in document doc-3.3 (PET). Layout has been adjusted slightly for readability and compactness.
  • Figure 4: Reflection example with Predicted and Gold standard constraints: (I) perfect match, (II) gold standard error, (III) ambiguous case