Failure-Driven Workflow Refinement
Jusheng Zhang, Kaitong Cai, Qinglin Zeng, Ningyuan Liu, Stephen Fan, Ziliang Chen, Keze Wang
TL;DR
This work reframes LLM workflow optimization as minimizing the Expected Failure Mass within a Failure Signature Space, preserving the structure of failures rather than collapsing them to binary success/failure signals. It introduces CE-Graph, a failure-driven refinement framework that clusters recurring failure modes from counterexamples and applies constrained graph edits via a Propose-and-Verify loop to greedily reduce failure mass. Empirically, CE-Graph delivers higher robustness at lower optimization cost across math, code, QA benchmarks and demonstrates transferability across models and datasets. The results suggest reliability emerges from learning and reshaping the geometry of failure distributions rather than simply avoiding failures, although the approach relies on detectable failure modes and incurs verification costs. Limitations include dependency on meaningful failure clustering and the need to carefully design operator libraries, with future work focusing on adaptive embeddings and risk-aware objectives.
Abstract
Optimizing LLM-based workflows is typically formulated as a global search, where candidate workflows are evaluated based on a scalar metric. This paradigm, however, suffers from a critical flaw: information collapse. By reducing rich, multi-step execution traces to simple success/failure signals, existing methods are rendered blind to the underlying structure of failures, fundamentally preventing them from modeling the workflow's failure distribution. We reconceptualize this challenge as a distributional problem. We propose a new paradigm where the optimization goal is not to maximize a scalar score, but to directly minimize a workflow's Expected Failure Mass, i.e., the integral of its failure probability density function defined over a high-dimensional Failure Signature Space (FSS). This distributional lens allows us to move from inefficient, zero-order optimization to a principled, gradient-like descent on the failure landscape itself. We introduce CE-Graph, a framework that operationalizes this paradigm through a novel, failure-driven refinement process. CE-Graph approximates the failure distribution from a pool of counterexamples, identifies its densest regions as recurring failure modes, and applies targeted, operator-constrained graph edits via a Propose-and-Verify mechanism to greedily reduce the failure mass. On math, code, and QA benchmarks, our CE-Graph achieves higher robustness at a significantly lower cost than strong baselines. This suggests that a system's reliability emerges not from avoiding failures, but from systematically learning and reshaping the geometric structure of its failure distributions.