Workflow-R1: Group Sub-sequence Policy Optimization for Multi-turn Workflow Construction

TL;DR

Workflow-R1 reframes automated workflow construction from static, one-shot code generation to a dynamic, multi-turn decision process where an LLM reasons in Think-Action cycles conditioned on running operators. Central to this is Group Sub-sequence Policy Optimization (GSsPO), which aligns gradient updates with the atomically meaningful Think-Action sub-sequences rather than tokens or whole sequences, providing a structure-aware credit assignment. The framework demonstrates state-of-the-art performance across seven QA benchmarks, with the GSsPO-based Workflow-R1 and its Search-augmented variant achieving superior exact-match results and exhibiting robust, self-correcting behavior grounded in execution feedback. The work advances autonomous agentic workflow optimization, offering a general RL approach to complex, dynamic reasoning tasks with practical implications for scalable, adaptable AI systems.

Abstract

The rapid evolution of agentic workflows has demonstrated strong performance of LLM-based agents in addressing complex reasoning tasks. However, existing workflow optimization methods typically formulate workflow synthesis as a static, one-shot code-centric generation problem. This paradigm imposes excessive constraints on the model's coding capabilities and restricts the flexibility required for dynamic problem-solving. In this paper, we present Workflow-R1, a framework that reformulates workflow construction as a multi-turn, natural language-based sequential decision-making process. To resolve the optimization granularity mismatch inherent in such multi-turn interactions, we introduce Group Sub-sequence Policy Optimization (GSsPO). While explicitly tailored to align with the interleaved Think-Action dynamics of agentic reasoning, GSsPO fundamentally functions as a structure-aware RL algorithm generalizable to a broad class of multi-turn agentic sequential decision-making tasks. By recalibrating the optimization unit to the composite sub-sequence, specifically the atomic Think-Action cycle, it aligns gradient updates with the semantic boundaries of these interactions, ensuring robust learning in complex multi-turn reasoning tasks. Through extensive experiments on multiple QA benchmarks, Workflow-R1 outperforms competitive baselines, validating GSsPO as a generalized solution for sequential reasoning and establishing Workflow-R1 as a promising new paradigm for automated workflow optimization.

Figures (5)

• Figure 1: Performance comparison of Workflow-R1 against SOTA baselines across seven benchmarks. Lighter and darker shades denote results using Qwen2.5-32B-Instruct and DeepSeek V3.2 as backbones, respectively. Workflow-R1 demonstrates superior performance across the evaluated benchmarks compared to both standard prompting strategies and advanced workflow optimization methods.
• Figure 2: The top panel shows conventional static workflow generation, where the agent produces a complete executable code-represented workflow. The bottom panel depicts Workflow-R1 multi-turn interaction, in which the agent incrementally constructs and adapts the workflow through think-action-observation cycles conditioned on execution results.
• Figure 2: Performance comparison of methods on QA benchmarks employing DeepSeek V3.2 as the execution model. This ablation evaluates the robustness of workflow optimization across different backbones. $^\dagger$ represents in-domain datasets.
• Figure 3: Mean reward convergence of GSsPO.
• Figure 4: Visualization of workflow construction for Workflow-R1-Search (left, blue) and Workflow-R1 (right, yellow).