A Ground Truth Approach for Assessing Process Mining Techniques

TL;DR

A ground-truth approach for generating process data from existing or synthetic initial process models, whether automatically generated or hand-made, which provides detailed insights into the strengths and weaknesses of process mining techniques, both quantitatively and qualitatively.

Abstract

The assessment of process mining techniques using real-life data is often compromised by the lack of ground truth knowledge, the presence of non-essential outliers in system behavior and recording errors in event logs. Using synthetically generated data could leverage ground truth for better evaluation. Existing log generation tools inject noise directly into the logs, which does not capture many typical behavioral deviations. Furthermore, the link between the model and the log, which is needed for later assessment, becomes lost. We propose a ground-truth approach for generating process data from either existing or synthetic initial process models, whether automatically generated or hand-made. This approach incorporates patterns of behavioral deviations and recording errors to produce a synthetic yet realistic deviating model and imperfect event log. These, together with the initial model, are required to assess process mining techniques based on ground truth knowledge. We demonstrate this approach to create datasets of synthetic process data for three processes, one of which we used in a conformance checking use case, focusing on the assessment of (relaxed) systemic alignments to expose and explain deviations in modeled and recorded behavior. Our results show that this approach, unlike traditional methods, provides detailed insights into the strengths and weaknesses of process mining techniques, both quantitatively and qualitatively.

Figures (14)

• Figure 1: Approaches to generating synthetic data (model $M$ and log $L$) for an underlying process $S$
• Figure 2: Abstract process model for the package delivery process, including modeled behavior of packages (black), FIFO queue (yellow), warehouse employees (green), delivery vans (orange), deliverers (blue), and depots (red).
• Figure 2: Deviation patterns.
• Figure 3: t-PNID modeling blueprints for three recording issue patterns (left) and four behavioral deviation patterns (right). Blue and green elements respectively correspond to matched and newly created elements.
• Figure 3: Experiment results showing the objects responsible for the deviation and those affected by it (in subscript) for each pattern, for the ground truth (GT) and detected by alignment methods: per-object ($\gamma^o$), systemic ($\gamma$), and relaxed systemic ($\tilde{\gamma}$).