Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowSteer: Interactive Agentic Workflow Orchestration via End-to-End Reinforcement Learning

Mingda Zhang, Haoran Luo, Tiesunlong Shen, Qika Lin, Xiaoying Tang, Rui Mao, Erik Cambria

TL;DR

FlowSteer tackles the challenge of scalable, reliable automated workflow orchestration by learning a policy that interacts with an executable canvas to construct, refine, and execute operator graphs. It introduces Canvas Workflow Relative Policy Optimization (CWRPO), which combines diversity-based structure rewards with conditional release of the task-correctness reward to stabilize long-horizon learning and prevent shortcut strategies. Across twelve benchmarks spanning QA, mathematics, and code, FlowSteer consistently outperforms static, offline, and other RL-based baselines, and demonstrates robust generalization across unseen tasks and backend models. The framework’s plug-and-play design enables seamless integration with diverse operator libraries and LLM backends, offering practical impact for complex, knowledge-intensive reasoning tasks.

Abstract

In recent years, a variety of powerful agentic workflows have been applied to solve a wide range of human problems. However, existing workflow orchestration still faces key challenges, including high manual cost, reliance on specific operators/large language models (LLMs), and sparse reward signals. To address these challenges, we propose FlowSteer, an end-to-end reinforcement learning framework that takes a lightweight policy model as the agent and an executable canvas environment, automating workflow orchestration through multi-turn interaction. In this process, the policy model analyzes execution states and selects editing actions, while the canvas executes operators and returns feedback for iterative refinement. Moreover, FlowSteer provides a plug-and-play framework that supports diverse operator libraries and interchangeable LLM backends. To effectively train this interaction paradigm, we propose Canvas Workflow Relative Policy Optimization (CWRPO), which introduces diversity-constrained rewards with conditional release to stabilize learning and suppress shortcut behaviors. Experimental results on twelve datasets show that FlowSteer significantly outperforms baselines across various tasks.

FlowSteer: Interactive Agentic Workflow Orchestration via End-to-End Reinforcement Learning

TL;DR

FlowSteer tackles the challenge of scalable, reliable automated workflow orchestration by learning a policy that interacts with an executable canvas to construct, refine, and execute operator graphs. It introduces Canvas Workflow Relative Policy Optimization (CWRPO), which combines diversity-based structure rewards with conditional release of the task-correctness reward to stabilize long-horizon learning and prevent shortcut strategies. Across twelve benchmarks spanning QA, mathematics, and code, FlowSteer consistently outperforms static, offline, and other RL-based baselines, and demonstrates robust generalization across unseen tasks and backend models. The framework’s plug-and-play design enables seamless integration with diverse operator libraries and LLM backends, offering practical impact for complex, knowledge-intensive reasoning tasks.

Abstract

In recent years, a variety of powerful agentic workflows have been applied to solve a wide range of human problems. However, existing workflow orchestration still faces key challenges, including high manual cost, reliance on specific operators/large language models (LLMs), and sparse reward signals. To address these challenges, we propose FlowSteer, an end-to-end reinforcement learning framework that takes a lightweight policy model as the agent and an executable canvas environment, automating workflow orchestration through multi-turn interaction. In this process, the policy model analyzes execution states and selects editing actions, while the canvas executes operators and returns feedback for iterative refinement. Moreover, FlowSteer provides a plug-and-play framework that supports diverse operator libraries and interchangeable LLM backends. To effectively train this interaction paradigm, we propose Canvas Workflow Relative Policy Optimization (CWRPO), which introduces diversity-constrained rewards with conditional release to stabilize learning and suppress shortcut behaviors. Experimental results on twelve datasets show that FlowSteer significantly outperforms baselines across various tasks.
Paper Structure (55 sections, 6 theorems, 70 equations, 7 figures, 10 tables)

This paper contains 55 sections, 6 theorems, 70 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology: FlowSteer
  5. Workflow Canvas and Agent Initialization
  6. Workflow Orchestration via Multi-Turn Interaction
  7. Outcome-Directed End-to-End Reinforcement Learning
  8. Experiments
  9. Experimental Setup
  10. Main Results (RQ1)
  11. Generalization Results (RQ2)
  12. Transferability across LLM Backends (RQ3)
  13. Ablation Study (RQ4)
  14. Comparison of RL Algorithms (RQ5)
  15. Conclusion
  16. ...and 40 more sections

Key Result

Proposition 1

The operator-action space $(\mathcal{O}, \mathcal{A})$ covers diverse workflow patterns across task types through canvas-grounded orchestration.

Figures (7)

  • Figure 1: Overview of the FlowSteer framework pipeline. The agent first initializes with the task and explores the search space. Then, through multi-turn interaction with the canvas, it analyzes workflow states, selects editing actions, and receives execution feedback to iteratively build and refine the workflow. Finally, the agent learns from diversity-constrained rewards to continuously improve its workflow orchestration strategies across diverse tasks.
  • Figure 2: Comparison of different workflow orchestration paradigms: static workflow selection, offline workflow generation, automated workflow optimization, and our interactive workflow orchestration framework FlowSteer.
  • Figure 3: An overview of the FlowSteer framework. The policy model (Flow-Director) interacts with Workflow Canvas through multi-turn interactions and learns from diversity-constrained rewards via CWRPO.
  • Figure 4: Transferability analysis of Flow-Director across LLM backends (RQ3). (a) Radar charts comparing six LLM backends (DeepSeek-V3.2, Grok-4.1-Fast, GPT-5.2, Claude-Opus-4.5, Gemini-3-Flash, Qwen-Plus) across six IID benchmarks, showing performance with and without Flow-Director trained on different backends. (b) Aggregated performance comparison across backends, grouped by task type (math, QA, code), comparing base LLMs vs. Flow-Director trained with 4o-mini vs. oss-120b. (c) Training dynamics showing F1 score, interaction turns, and operator counts over training steps for different backend configurations.
  • Figure 5: Ablation and RL algorithm analysis. (a) Token consumption comparison across task types, showing FlowSteer achieves lower token usage. (b) Average interaction turns comparison, demonstrating FlowSteer requires fewer turns to complete tasks. (c-e) Pairwise performance comparison matrices for math, QA, and code tasks respectively, where positive values (red) indicate row method outperforms column method.
  • ...and 2 more figures

Theorems & Definitions (12)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • Proposition 4: Operator-Action Cognitive Completeness
  • proof
  • Proposition 5: Monotonic Improvement of Multi-turn Interaction
  • proof
  • ...and 2 more