WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

TL;DR

WorkflowPerturb addresses the calibration gap in evaluating LLM-generated workflows by introducing a controlled perturbation benchmark: 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities $10\%$, $30\%$, and $50\%$. It evaluates seven metric families (structural, lexical, semantic, ordering, and judgment-based) and a GPT-4o-based LLM-as-Judge, revealing distinct sensitivity and calibration patterns across perturbations and metrics. The findings show that no single metric captures all failure modes, underscoring the need for severity-aware interpretation and complementary metric bundles for CI/CD validation in production. The dataset and insights enable practitioners to calibrate metrics and select robust bundles before deployment, with dataset release upon acceptance.

Abstract

LLM-based systems increasingly generate structured workflows for complex tasks. In practice, automatic evaluation of these workflows is difficult, because metric scores are often not calibrated, and score changes do not directly communicate the severity of workflow degradation. We introduce WorkflowPerturb, a controlled benchmark for studying workflow evaluation metrics. It works by applying realistic, controlled perturbations to golden workflows. WorkflowPerturb contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, and Description Changes), each applied at severity levels of 10%, 30%, and 50%. We benchmark multiple metric families and analyze their sensitivity and calibration using expected score trajectories and residuals. Our results characterize systematic differences across metric families and support severity-aware interpretation of workflow evaluation scores. Our dataset will be released upon acceptance.

Figures (6)

• Figure 1: Real-world example illustrating perturbations in LLM-generated workflows for a customer support escalation task. Missing Steps: The red node, present in the original workflow, is omitted in the LLM-generated version. Compressed Steps: The red and blue nodes from the original workflow are merged into a single blue node, reducing workflow granularity. Description Change: All nodes retain the same semantic meaning as the original but differ in syntactic phrasing.
• Figure 2: Visualization of Missing Steps perturbations applied to the six-node blog generation workflow. At each level, a proportional number of nodes is randomly dropped: 1 node (10%), 2 nodes (30%), and 3 nodes (50%). Higher perturbation levels result in increasingly incomplete workflows that diverge from the golden sequence.
• Figure 3: Metric scores across perturbation levels (mean $\pm$ std) for Missing Steps.
• Figure 4: Metric scores across perturbation levels (mean $\pm$ std) for Compressed Steps.
• Figure 5: Metric scores across perturbation levels (mean $\pm$ std) for Description Changes.