Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks

TL;DR

The work reframes failures in long-horizon language-grounded tasks as reliability failures caused by stochastic drift from a canonical solution path rather than mere capability gaps. It defines an empirical canonical path via consensus tool sets and demonstrates, through a natural experiment in Toolathlon, that successful runs adhere more closely to this path than failed runs, with a within-unit adherence gap of +$0.060$ in Jaccard ($p<0.0001$). The drift is gradual and self-reinforcing, ruling out early-branching explanations and enabling mid-trajectory interventions; a monitor restarting the bottom tercile by 75% completion yields an approximate +$8.8$ percentage-point lift in success. Across model families and tasks, the canonical-path mechanism generalizes, motivating reliability-aware benchmarking and mid-trajectory scaffolding as practical deployment strategies to improve real-world agent performance.

Abstract

Why do language agents fail on tasks they are capable of solving? We argue that many such failures are reliability failures caused by stochastic drift from a task's latent solution structure, not capability failures. Every well-defined tool-use task imposes a canonical solution path (i.e., a convergent set of tool invocations shared across successful runs) and agent success depends critically on whether a trajectory stays within this path's operating envelope. We establish this causally using a natural experiment that holds model capability and task difficulty fixed by construction. We analyze trajectories from the Toolathlon benchmark: 22 frontier models each attempt 108 real-world tool-use tasks across 3 independent runs, yielding 515 model$\times$task units where the same model succeeds on some runs and fails on others due to LLM sampling stochasticity alone. Within these units, successful runs adhere significantly more closely to the canonical solution path than failed runs ($+$0.060 Jaccard, $p<0.0001$, $n=488$ units, 95% CI [+0.043, +0.077]). This result survives six robustness checks including cross-model-family leave-one-out validation. Critically, the causal mechanism is gradual and self-reinforcing: the adherence gap is statistically indistinguishable from zero through the first 50% of the trajectory, ruling out early-branching selection bias, and each off-canonical tool call raises the probability that the next call is also off-canonical by 22.7 percentage points ($\hatβ=+0.227$, $p<0.0001$), more than doubling the baseline rate. These findings imply that agent reliability cannot be improved by capability scaling alone, but offer a highly actionable intervention: a simple monitor that restarts the bottom tercile of runs based on mid-trajectory canonical adherence lifts success rates by $+$8.8 percentage points among intervened runs.

Figures (4)

• Figure 1: Canonical path drift is gradual and self-reinforcing.(A) Within mixed-outcome units ($n=488$), the adherence gap between successful and failed runs is statistically indistinguishable from zero at 10%, 25%, and 50% of trajectory completion, becoming significant only at 75% ($p = 0.043$) and strongly significant at full completion ($p < 0.0001$), ruling out early branching as the mechanism. (B) Conditional transition probabilities show that off-canonical calls are self-reinforcing: following an off-canonical call, the probability that the next call is also off-canonical rises by $+22.7$pp relative to following a canonical call ($\hat{\beta}=+0.227$, $p<0.0001$), and this compounding is significantly stronger in runs that ultimately fail. Together, panels A and B establish that agent failure is a gradual compound process, not a single pivotal mistake.
• Figure 2: Robustness and generalization of the canonical path adherence effect.(A) Within-unit adherence gaps across 70 tasks; 56/70 (80%) show positive effects (mean $= +0.068$, green dashed line). (B) Effect size is larger for shorter tasks ($r = -0.25$, $p = 0.038$), consistent with less opportunity for course-correction after deviation; colour indicates canonical path strength. (C) All 11 model families show positive gaps; 5 reach individual significance ($p < 0.05$, dark blue). Error bars show $\pm 1.96$ SE.
• Figure 3: Sample construction from full dataset to CF-LOO analysis sample. The less conservative LOO specification retains $n=495$ units; results are substantively identical.
• Figure 4: Trajectory comparison for git-milestone (glm-4.6). Both runs share an identical four-step prefix (shaded); Run 2 then completes canonically in 2 further steps, while Run 1 deviates at step 6 and spirals through 32 off-canonical calls without reaching write_file. Within-unit adherence gap: $+0.694$ Jaccard (sample mean: $+0.060$).