Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement

TL;DR

The paper tackles extracting executable procedural graphs from natural language by introducing text2flow, a multi-agent framework that enforces structural validity and semantic alignment via external, natural-language feedback. It decomposes the task into three stages—graph construction (Graph Builder), structural validation (Simulation), and logical alignment (Semantic)—and refines graphs through iterative, feedback-driven prompting with a prioritized, budgeted revision mechanism. Empirical results on the PAGED benchmark show substantial improvements in both structural correctness and logical consistency over strong baselines, with a detailed analysis of efficiency and token costs. The work demonstrates the value of an interpretable evaluation sandbox for guided, multi-round procedural graph generation and points to extensions toward broader workflow understanding tasks.

Abstract

Automatically extracting workflows as procedural graphs from natural language is promising yet underexplored, demanding both structural validity and logical alignment. While recent large language models (LLMs) show potential for procedural graph extraction, they often produce ill-formed structures or misinterpret logical flows. We present \model{}, a multi-agent framework that formulates procedural graph extraction as a multi-round reasoning process with dedicated structural and logical refinement. The framework iterates through three stages: (1) a graph extraction phase with the graph builder agent, (2) a structural feedback phase in which a simulation agent diagnoses and explains structural defects, and (3) a logical feedback phase in which a semantic agent aligns semantics between flow logic and linguistic cues in the source text. Important feedback is prioritized and expressed in natural language, which is injected into subsequent prompts, enabling interpretable and controllable refinement. This modular design allows agents to target distinct error types without supervision or parameter updates. Experiments demonstrate that \model{} achieves substantial improvements in both structural correctness and logical consistency over strong baselines.

Figures (4)

• Figure 1: Overview of the text2flow multi-agent framework. The Graph Builder Agent generates an initial procedural graph from the document. The Evaluation Sandbox, with simulation and semantic agents, identifies structural and logic errors. Prioritized revisions are iteratively applied to the Graph Builder Agent until a valid graph is produced.
• Figure 2: Ablation Study on GPT-4o backbone. Each bar shows the performance under different module removals: w/o-sandbox – few-shot generation without evaluation sandbox; w/o-simulator – iterative refinement using only the semantic agent as feedback; w/o-semantic – iterative refinement using only the simulator as feedback; w/o-prioritizing – all revision candidates are sent to the graph builder without filtering. text2flow denotes the full model with all components enabled.
• Figure 3: F1-scores and average time per sample across iterations (0 = few-shot). Gains peak around 2 iterations, after which improvements plateau while cost increases.
• Figure 4: F1-score comparison of Zero-Shot, Few-Shot, and text2flow on GPT-4o and Qwen3:30B. text2flow shows consistent gains across all categories.