Aegis: Automated Error Generation and Attribution for Multi-Agent Systems

TL;DR

The paper addresses the scarcity of large, diverse datasets for attributing failures in multi-agent systems (MAS) and the resulting debugging challenges. It introduces Aegis, a fully automated pipeline that generates 9,533 annotated error trajectories by injecting context-aware faults into successful MAS executions, across six frameworks and six task domains. Aegis supports three learning paradigms—Supervised Fine-Tuning (SFT), Reinforcement Learning with Group Relative Policy Optimization (GRPO), and Disentangled Contrastive Learning (DCL)—and demonstrates substantial gains in error attribution on in-domain and out-of-distribution benchmarks, with fine-tuned models rivaling much larger proprietary systems. The work advances MAS reliability by providing a scalable data-engineering solution, validated across multiple tasks and architectures, and open-sourcing data, code, and models to enable further research into robust and interpretable agentic systems.

Abstract

Large language model based multi-agent systems (MAS) have unlocked significant advancements in tackling complex problems, but their increasing capability introduces a structural fragility that makes them difficult to debug. A key obstacle to improving their reliability is the severe scarcity of large-scale, diverse datasets for error attribution, as existing resources rely on costly and unscalable manual annotation. To address this bottleneck, we introduce Aegis, a novel framework for Automated error generation and attribution for multi-agent systems. Aegis constructs a large dataset of 9,533 trajectories with annotated faulty agents and error modes, covering diverse MAS architectures and task domains. This is achieved using a LLM-based manipulator that can adaptively inject context-aware errors into successful execution trajectories. Leveraging fine-grained labels and the structured arrangement of positive-negative sample pairs, Aegis supports three different learning paradigms: Supervised Fine-Tuning, Reinforcement Learning, and Contrastive Learning. We develop learning methods for each paradigm. Comprehensive experiments show that trained models consistently achieve substantial improvements in error attribution. Notably, several of our fine-tuned LLMs demonstrate performance competitive with or superior to proprietary models an order of magnitude larger, validating our automated data generation framework as a crucial resource for developing more robust and interpretable multi-agent systems. Our project website is available at https://kfq20.github.io/Aegis-Website/.

Figures (5)

• Figure 1: An overview of the Aegis framework. The Data Construction pipeline (left) automatically generates a dataset of labeled failures by taking successful multi-agent trajectories and applying controlled, context-aware error injections via an adaptive manipulator. The resulting dataset's structure enables three distinct Learning Methods (right) for the error attribution task.
• Figure 2: Performance (average score) of different models on Aegis-Bench, broken down by task domain (left) and MAS framework (right).
• Figure 3: Analysis of GRPO and SFT training, showing (a) GRPO reward curves, (b) the influence of Chain-of-Thought (CoT) prompting on performance, and (c) the trade-off between in-domain (Aegis-Bench) and out-of-distribution (Who&When) performance across SFT epochs.
• Figure 4: A t-SNE visualization of the learned embedding space at the trajectory (bag) and individual turn levels.
• Figure 5: A case study of Who&When in a financial planning task. Aegis correctly identifies the root-cause error made by the Tickets_Pricing_Expert, while baseline models either misattribute the failure to a downstream agent or fail to detect any error at all.