WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models

TL;DR

The experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs and exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval.

Abstract

Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration. It first constructs a large-scale fine-tuning dataset WorkflowBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. Specifically, the construction process can be divided into three phases: (1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code. We further equip them with generated hierarchical thought via ChatGPT. (2) Query Expansion: we prompt ChatGPT to generate more task queries to enrich the diversity and complexity of workflows. (3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench. Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama. Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval. Our data and code are available at https://github.com/OpenBMB/WorkflowLLM.

Figures (7)

• Figure 1: Overview of WorkflowLLM. It first constructs WorkflowBench through a three-phase pipeline and fine-tunes WorkflowLlama, which can generate workflows based on the user's query (appointment management in this case).
• Figure 2: Illustration of our WorkflowLLM which contains three phases to construct WorkflowBench, followed by the supervised fine-tuning phase to derive WorkflowLlama.
• Figure 3: Illustration of data field composition in WorkflowBench comprising Task Query, API documentations, Task Plan, and Workflow code with Comments.
• Figure 4: Comparison of the distributions across workflow categories, APPs, and action counts in the collected data and the final dataset. The upper section shows the original data collected from Apple Shortcuts and RoutineHub, while the lower section presents the expanded dataset distributions.
• Figure 5: Performance comparisons based on the number of actions, the number of Branch & Loop, and the nested depth of the reference code.