Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention

TL;DR

The paper addresses the problem that execution-time interventions by binary LLM critics may not reliably improve deployment reliability. It introduces a disruption–recovery framework, formalizing a threshold $p^*=d/(r+d)$ that determines whether intervention is beneficial, and demonstrates that high offline accuracy (AUROC ≈ 0.94) does not guarantee positive outcomes due to the agent’s ability to absorb mid-trajectory corrections. Through experiments across HotPotQA, GAIA, and ALFWorld with simple intervention mechanisms, it shows that interventions often harm high-success systems and yield only modest gains in low-success regimes, with gains bounded by an intrinsic disruption cost. A 50-task pilot test before deployment accurately forecasts regime-specific outcomes and supports a practical deployment guideline: intervene only when the pilot indicates a favorable disruption–recovery balance; avoid early-step interventions and favor post-hoc selection when disruption dominates recovery. The work concludes with deployment guidelines, ablations, and limitations, emphasizing that failure prediction is necessary but not sufficient for safe failure prevention, and that practitioners should treat intervention as a model-dependent control problem rather than a pure prediction task.

Abstract

Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentage point (pp) collapse on one model while affecting another by near zero pp. This variability demonstrates that LLM critic accuracy alone is insufficient to determine whether intervention is safe. We identify a disruption-recovery tradeoff: interventions may recover failing trajectories but also disrupt trajectories that would have succeeded. Based on this insight, we propose a pre-deployment test that uses a small pilot of 50 tasks to estimate whether intervention is likely to help or harm, without requiring full deployment. Across benchmarks, the test correctly anticipates outcomes: intervention degrades performance on high-success tasks (0 to -26 pp), while yielding a modest improvement on the high-failure ALFWorld benchmark (+2.8 pp, p=0.014). The primary value of our framework is therefore identifying when not to intervene, preventing severe regressions before deployment.

Figures (1)

• Figure 1: Decision tree illustrating the recommended deployment procedure for execution-time intervention, based on pilot estimates of failure ($p$), recovery ($r$), and disruption ($d$), and the resulting threshold $p^\star = d/(r+d)$ (left). Example calculations for ALFWorld with Qwen-3-8B are shown on the right.