Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProcessTBench: An LLM Plan Generation Dataset for Process Mining

Andrei Cosmin Redis, Mohammadreza Fani Sani, Bahram Zarrin, Andrea Burattin

TL;DR

The ProcessTBench synthetic dataset is presented, an extension of the TaskBench dataset specifically designed to evaluate LLMs within a process mining framework.

Abstract

Large Language Models (LLMs) have shown significant promise in plan generation. Yet, existing datasets often lack the complexity needed for advanced tool use scenarios - such as handling paraphrased query statements, supporting multiple languages, and managing actions that can be done in parallel. These scenarios are crucial for evaluating the evolving capabilities of LLMs in real-world applications. Moreover, current datasets don't enable the study of LLMs from a process perspective, particularly in scenarios where understanding typical behaviors and challenges in executing the same process under different conditions or formulations is crucial. To address these gaps, we present the ProcessTBench synthetic dataset, an extension of the TaskBench dataset specifically designed to evaluate LLMs within a process mining framework.

ProcessTBench: An LLM Plan Generation Dataset for Process Mining

TL;DR

The ProcessTBench synthetic dataset is presented, an extension of the TaskBench dataset specifically designed to evaluate LLMs within a process mining framework.

Abstract

Large Language Models (LLMs) have shown significant promise in plan generation. Yet, existing datasets often lack the complexity needed for advanced tool use scenarios - such as handling paraphrased query statements, supporting multiple languages, and managing actions that can be done in parallel. These scenarios are crucial for evaluating the evolving capabilities of LLMs in real-world applications. Moreover, current datasets don't enable the study of LLMs from a process perspective, particularly in scenarios where understanding typical behaviors and challenges in executing the same process under different conditions or formulations is crucial. To address these gaps, we present the ProcessTBench synthetic dataset, an extension of the TaskBench dataset specifically designed to evaluate LLMs within a process mining framework.
Paper Structure (11 sections, 2 figures, 2 tables)

This paper contains 11 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Generating ProcessTBench
  3. Description of ProcessTBench
  4. Queries
  5. Ground Truth and Generated Plans
  6. Use-cases
  7. Evaluating Plan Generation by LLMs:
  8. Evaluating Paraphrase Handling in Plan Generation by LLMs:
  9. Using Process Mining Algorithms for Plan Generation:
  10. Plan Generation Variability and Reliability by LLMs:
  11. Conclusion

Figures (2)

  • Figure 1: The data generation pipeline of ProcessTBench. The arrows symbolize that the output of one step is input to the next.
  • Figure 2: Key quality features of the queries in ProcessTBench.