DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems

TL;DR

The paper addresses debugging failures in LLM-based multi-agent systems where errors span long interaction traces. It introduces DoVer, an intervention-driven do-then-verify framework that segments failure traces into trials, generates targeted interventions on plan or messages, and replays to verify outcomes. Empirically, it demonstrates that DoVer recovers a substantial portion of failures and enables explicit validation or refutation of failure hypotheses across GAIA/AssistantBench and GSMPlus, highlighting improved reliability and an outcome-oriented debugging paradigm. The work also analyzes ground-truth uncertainty in log-based attribution and discusses generalizability to other agent frameworks, data regimes, and potential future directions for automated sub-agent repair and richer intervention spaces.

Abstract

Large language model (LLM)-based multi-agent systems are challenging to debug because failures often arise from long, branching interaction traces. The prevailing practice is to leverage LLMs for log-based failure localization, attributing errors to a specific agent and step. However, this paradigm has two key limitations: (i) log-only debugging lacks validation, producing untested hypotheses, and (ii) single-step or single-agent attribution is often ill-posed, as we find that multiple distinct interventions can independently repair the failed task. To address the first limitation, we introduce DoVer, an intervention-driven debugging framework, which augments hypothesis generation with active verification through targeted interventions (e.g., editing messages, altering plans). For the second limitation, rather than evaluating on attribution accuracy, we focus on measuring whether the system resolves the failure or makes quantifiable progress toward task success, reflecting a more outcome-oriented view of debugging. Within the Magnetic-One agent framework, on the datasets derived from GAIA and AssistantBench, DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses. DoVer also performs effectively on a different dataset (GSMPlus) and agent framework (AG2), where it recovers 49% of failed trials. These results highlight intervention as a practical mechanism for improving reliability in agentic systems and open opportunities for more robust, scalable debugging methods for LLM-based multi-agent systems. Project website and code will be available at https://aka.ms/DoVer.

Figures (12)

• Figure 1: Failure trace of Case 3 in WW-HC, illustrating ambiguity in failure attribution. The session consists of four distinct trials, each initiated by a plan update and executed via a ReAct-style React loop. Different strategies (e.g., direct scrolling in Trial 1 vs. calendar navigation in Trial 2) yield separate error points, making single-step attribution across the session inherently ambiguous. Trial 2 (Steps 53–55) further shows inter-agent misalignment: the Orchestrator issued an invalid instruction, while the WebSurfer compounded the error by executing an unrelated action.
• Figure 2: DoVer (Do--then--Verify) Debugging Pipeline. (1) Trial segmentation: split the failed session log into trials using re-plan steps as cut points. (2) Failure attribution: for each trial, propose a hypothesis $h_i$ that marks a faulty step or agent. (3) Intervention generation: turn $h_i$ into an actionable intervention that edits either the plan or the attributed message or step in the original log. (4) Intervention execution: replay the trajectory in place, i.e., preserve all steps before the intervened step, then execute the intervention and measure progress of the new log. Colors indicate plan/re-plan (blue), execution (green), attributed failure (red), terminal failure (dark red), terminal success (dark green), intervention (yellow), new plan/re-plan (blue hatch), and new execution (green hatch).
• Figure 3: Extended prompt template: orange marks explicit step indices, and blue marks the embedded concise reminder of annotators' guidance.
• Figure 5: Tagging results for each GAIA case in WW. The table reports ground-truth annotations, uncertainty tags, possibility for multi-failure step attribution, presence of ambiguous attributions, potential API or flaky errors, and case-specific details such as number of trials and model outputs. Model predictions matching the ground-truth failure step are highlighted in green.
• Figure 6: Web-based intervention user interface for the AG2 MathChat system. (1) List of recorded math problem sessions. (2) Input box for submitting a new math task. (3) Main conversation panel showing the multi-agent trace and intermediate reasoning. (4) Intervention panel, where a user can select a specific agent and step to edit the message or plan. (5) History panel showing checkpoints and continuations after interventions.