From Observability Data to Diagnosis: An Evolving Multi-agent System for Incident Management in Cloud Systems

TL;DR

This work introduces OpsAgent, a lightweight, self-evolving multi-agent system for incident management in large-scale cloud environments. It tackles heterogeneity in observability data with a training-free data processor, enables transparent and auditable reasoning through role-based collaboration, and achieves continual improvement via a dual self-evolution mechanism that blends PPO updates with reflection-based knowledge distillation. Evaluations on the OPENRCA benchmark demonstrate state-of-the-art performance, strong generalization, interpretability, and cost efficiency, supporting practical deployment in real-world cloud systems. The approach yields a sustainable, auditable, and adaptable IM solution that can generalize to other domains dealing with massive, heterogeneous data.

Abstract

Incident management (IM) is central to the reliability of large-scale cloud systems. Yet manual IM, where on-call engineers examine metrics, logs, and traces is labor-intensive and error-prone in the face of massive and heterogeneous observability data. Existing automated IM approaches often struggle to generalize across systems, provide limited interpretability, and incur high deployment costs, which hinders adoption in practice. In this paper, we present OpsAgent, a lightweight, self-evolving multi-agent system for IM that employs a training-free data processor to convert heterogeneous observability data into structured textual descriptions, along with a multi-agent collaboration framework that makes diagnostic inference transparent and auditable. To support continual capability growth, OpsAgent also introduces a dual self-evolution mechanism that integrates internal model updates with external experience accumulation, thereby closing the deployment loop. Comprehensive experiments on the OPENRCA benchmark demonstrate state-of-the-art performance and show that OpsAgent is generalizable, interpretable, cost-efficient, and self-evolving, making it a practically deployable and sustainable solution for long-term operation in real-world cloud systems.

Figures (7)

• Figure 1: From lightweight LLM to MAS-based IM. OpsAgent turns a lightweight LLM into a deployable and sustainable IM system by incorporating (1) training-free data processor (Section \ref{['sec:data_processor']}), (2) multi-agent collaboration (Section \ref{['sec:multi_agent']}), and (3) self-evolution mechanism (Section \ref{['sec:self-evolution']}).
• Figure 2: Training-free Data Processor. The processor handles three types of observability data separately: metrics (left), logs (middle), and traces (right).
• Figure 3: Illustrative example of data descriptions.
• Figure 4: Multi-agent Collaboration. Agents with predefined roles (via agent profile) cooperate under a structured workflow and cross-review mechanism to enhance reasoning from multiple perspectives. The Root Cause Report not only guides online incident mitigation but also feeds offline training, closing the loop for sustainable capability growth.
• Figure 5: Self-evolution Mechanism. Internally, agents are fine-tuned via PPO training with a carefully designed reward model (top). Externally, a reflection process distills reusable knowledge into a task-specific knowledge base, which is later leveraged through RAG for knowledge injection (bottom).