Skill Learning Using Process Mining for Large Language Model Plan Generation

TL;DR

A novel approach to skill learning in LLMs is introduced by integrating process mining techniques, leveraging process discovery for skill acquisition, process models for skill storage, and conformance checking for skill retrieval, which suggests the effectiveness of the approach.

Abstract

Large language models (LLMs) hold promise for generating plans for complex tasks, but their effectiveness is limited by sequential execution, lack of control flow models, and difficulties in skill retrieval. Addressing these issues is crucial for improving the efficiency and interpretability of plan generation as LLMs become more central to automation and decision-making. We introduce a novel approach to skill learning in LLMs by integrating process mining techniques, leveraging process discovery for skill acquisition, process models for skill storage, and conformance checking for skill retrieval. Our methods enhance text-based plan generation by enabling flexible skill discovery, parallel execution, and improved interpretability. Experimental results suggest the effectiveness of our approach, with our skill retrieval method surpassing state-of-the-art accuracy baselines under specific conditions.

Figures (5)

• Figure 1: In the skill learning approach, when given a prompt such as 'Arrange my meeting tomorrow with John,' the LLM plan generator retrieves the 'meeting' skill to enhance its response. This paper introduces process mining techniques to discover this skill in a process model format, facilitating its retrieval and offering additional advantages, such as enabling parallelism in plan execution.
• Figure 2: Schematic view of the skill learning process using process discovery.
• Figure 3: Summary of the overall accuracy of the traces generated by the planner for n=2173 cases.
• Figure 4: performance of the proposed ada-002 and conformance hybrid model, given different planner accuracies. The columns represent different planner accuracies, while the rows represent the metric, f1, or MRR. The x-axis represents the nearest neighbors retrieved by the first stage retrieval model, ada-002. The y-axis represents the final result of the respective metric after reranking with Conformance Checking. The reference lines show the results at the baseline planner accuracy (fitness) threshold = 0.0. The figure reveals that with a counterfactual conformance checking threshold of 0.7 alignment fitness, the combined ada-002 and Conformance Checking would outperform all other methods.
• Figure :