Do We Always Need Query-Level Workflows? Rethinking Agentic Workflow Generation for Multi-Agent Systems

TL;DR

The paper investigates the necessity of query-level workflow generation in large-language-model–driven multi-agent systems and reveals that a small set of top task-level workflows can achieve as good or better query coverage while avoiding the high cost of per-query generation. It demonstrates that exhaustive execution-based evaluation at the task level is expensive and often unreliable, motivating SCALE, a low-cost framework that replaces full validation with self-prediction plus few-shot calibration to estimate workflow quality. Across six diverse benchmarks, SCALE achieves comparable performance to strong baselines while reducing token usage by up to 83%, validating its practicality for scalable MAS deployment. The work provides both empirical insights and a concrete methodology for cost-efficient MAS workflow synthesis, with ablations showing calibrated surrogate scores offer robust guidance for search and ranking.

Abstract

Multi-Agent Systems (MAS) built on large language models typically solve complex tasks by coordinating multiple agents through workflows. Existing approaches generates workflows either at task level or query level, but their relative costs and benefits remain unclear. After rethinking and empirical analyses, we show that query-level workflow generation is not always necessary, since a small set of top-K best task-level workflows together already covers equivalent or even more queries. We further find that exhaustive execution-based task-level evaluation is both extremely token-costly and frequently unreliable. Inspired by the idea of self-evolution and generative reward modeling, we propose a low-cost task-level generation framework \textbf{SCALE}, which means \underline{\textbf{S}}elf prediction of the optimizer with few shot \underline{\textbf{CAL}}ibration for \underline{\textbf{E}}valuation instead of full validation execution. Extensive experiments demonstrate that \textbf{SCALE} maintains competitive performance, with an average degradation of just 0.61\% compared to existing approach across multiple datasets, while cutting overall token usage by up to 83\%.

Figures (4)

• Figure 1: Comparison of Aflow and our rethought task-level workflow generation framework. Left: total token number during workflow generation (log-scale axis). Right: final test performance. Our method SCALE achieves comparable performance while significantly reducing token number.
• Figure 2: Task-level vs. Query-level workflow generation on their process and rethinking.(1)Task-level generation.(1)A shows searching/training: the generator generates a single workflow using validation queries; (1)B shows inference: the optimized workflow is reused for all test queries. (1)C–D present our rethinking: (1)C shows that repeated full-set evaluations is very token-costly, and (1)D shows top-k workflows have very similar query-level ranks. (2)Query-level generation. (2)A shows training: a workflow is generated per query; (2)B shows inference: producing customized workflows for each input. (2)C1–C3 summarize our rethinking on query-level workflows: top-k task-level workflows, repeat-k runs of the top-1 workflow, and true query-level generation yield comparable coverage/performance.
• Figure 3: Cumulative Token Number v.s. Performance during Aflow's task-level workflow generation process.
• Figure 4: Performance and ranking statistics of the top-$5$ task-level workflows generated by Aflow across four benchmarks. Perf denotes average test performance. CR and DR denote average competition rank and dense rank, respectively, computed over test queries.