StarFlow: Generating Structured Workflow Outputs From Sketch Images

TL;DR

StarFlow tackles the problem of converting sketch-style workflow diagrams into executable structured outputs. It proposes finetuning vision-language models on a diverse, synthetic-to-real dataset of workflow diagrams and evaluates outputs with a tree-structured FlowSim metric, including a formal FlowSim = $1 - TED(F, F_r)/( |F| + |F_r| )$ and TreeBLEU alongside Trigger/Component matches. The results show that finetuning substantially improves structured workflow generation, often surpassing general-purpose models and approaching or exceeding large proprietary baselines on this task. The work provides an open-source pipeline, datasets, and evaluation tooling, highlighting practical implications for accessible, diagram-driven workflow automation while outlining safety, grounding, and generalization considerations for real-world deployment.

Abstract

Workflows are a fundamental component of automation in enterprise platforms, enabling the orchestration of tasks, data processing, and system integrations. Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. To simplify this process, we explore the use of generative foundation models, particularly vision-language models (VLMs), to automatically generate structured workflows from visual inputs. Translating hand-drawn sketches or computer-generated diagrams into executable workflows is challenging due to the ambiguity of free-form drawings, variations in diagram styles, and the difficulty of inferring execution logic from visual elements. To address this, we introduce StarFlow, a framework for generating structured workflow outputs from sketches using vision-language models. We curate a diverse dataset of workflow diagrams -- including synthetic, manually annotated, and real-world samples -- to enable robust training and evaluation. We finetune and benchmark multiple vision-language models, conducting a series of ablation studies to analyze the strengths and limitations of our approach. Our results show that finetuning significantly enhances structured workflow generation, outperforming large vision-language models on this task.

Figures (16)

• Figure 1: The task of sketch to workflow. Given an input image representing a business process, the task is to convert the logic of the diagram found in the image into a structured JSON output describing the execution logic of the workflow, including the appropriate trigger, actions, and inputs.
• Figure 2: Performance of each model per type of sample. We report the FlowSim with inputs results. Number of supporting examples for each sample type is shown in parenthesis. Examples for each type of sample can be found in Appendix \ref{['sec:appendix_sample_types']}.
• Figure 3: Impact of image orientation. We report the FlowSim with inputs results. Number of supporting examples for each sample type is shown in parenthesis.
• Figure 4: Impact of image resolution. We report the FlowSim with input results. The number of supporting examples for each sample type is shown in parentheses.
• Figure 5: Screenshot of a workflow. The screenshot was taken from a visualization platform for which we don't have examples in the training dataset.