JudgeFlow: Agentic Workflow Optimization via Block Judge

TL;DR

JudgeFlow introduces an Evaluation-Judge-Optimization-Update pipeline that augments code-represented agentic workflows with reusable logic blocks (seq, for, cond) and a dedicated Judge to assign rank-based responsibility to underperforming blocks. This fine-grained diagnostic signal guides a targeted, LLM-based optimizer to modify the most problematic block, improving sample efficiency and interpretability. Across GSM8K, MATH, MBPP, HumanEval, and AIME 2025 benchmarks, JudgeFlow outperforms strong baselines and demonstrates robust cross-benchmark generalization. The approach offers a scalable framework for automating increasingly complex agentic workflows, albeit with a caveat regarding potential judge biases in LLMs that warrant further robustness enhancements.

Abstract

Optimizing LLM-based agentic workflows is challenging for scaling AI capabilities. Current methods rely on coarse, end-to-end evaluation signals and lack fine-grained signals on where to refine, often resulting in inefficient or low-impact modifications. To address these limitations, we propose {\our{}}, an Evaluation-Judge-Optimization-Update pipeline. We incorporate reusable, configurable logic blocks into agentic workflows to capture fundamental forms of logic. On top of this abstraction, we design a dedicated Judge module that inspects execution traces -- particularly failed runs -- and assigns rank-based responsibility scores to problematic blocks. These fine-grained diagnostic signals are then leveraged by an LLM-based optimizer, which focuses modifications on the most problematic block in the workflow. Our approach improves sample efficiency, enhances interpretability through block-level diagnostics, and provides a scalable foundation for automating increasingly complex agentic workflows. We evaluate {\our{}} on mathematical reasoning and code generation benchmarks, where {\our{}} achieves superior performance and efficiency compared to existing methods. The source code is publicly available at https://github.com/ma-zihan/JudgeFlow.

Figures (6)

• Figure 1: Block-level judging guides agentic workflow optimization by identifying the most problematic block in failed executions.
• Figure 2: The illustration of logic blocks.
• Figure 3: The main pipeline of JudgeFlow
• Figure 4: Performance on AIME 2025.
• Figure 5: \ref{['fig:ab1']} The optimal workflow found by JudgeFlow on MBPP dataset; \ref{['fig:ab2']} The training and testing curve between JudgeFlow and AFlow on MBPP dataset.