ExOAR: Expert-Guided Object and Activity Recognition from Textual Data

TL;DR

ExOAR tackles the problem of extracting structured object-centric logs from unstructured textual traces by coupling LLM-based semantic recognition with expert validation in a four-step pipeline. The approach iteratively refines object types, activities, and object instances before enriching events for OCEL-based process mining. Demonstrations on Active Window Tracking data and preliminary multi-domain evaluation show the method is feasible, flexible, and reduces manual labeling burden while preserving semantic fidelity. The work advances practical semantic interpretation of workplace text and supports scalable process mining across domains that produce textual traces.

Abstract

Object-centric process mining requires structured data, but extracting it from unstructured text remains a challenge. We introduce ExOAR (Expert-Guided Object and Activity Recognition), an interactive method that combines large language models (LLMs) with human verification to identify objects and activities from textual data. ExOAR guides users through consecutive stages in which an LLM generates candidate object types, activities, and object instances based on contextual input, such as a user's profession, and textual data. Users review and refine these suggestions before proceeding to the next stage. Implemented as a practical tool, ExOAR is initially validated through a demonstration and then evaluated with real-world Active Window Tracking data from five users. Our results show that ExOAR can effectively bridge the gap between unstructured textual data and the structured log with clear semantics needed for object-centric process analysis, while it maintains flexibility and human oversight.

Figures (2)

• Figure 1: Overview of our approach, depicting a User (U) and ExOAR (E).
• Figure 2: Screenshot of Step 1 - Object Identification