MicroRemed: Benchmarking LLMs in Microservices Remediation

TL;DR

MicroRemed introduces the first end-to-end benchmark for LLM-based microservice remediation, requiring models to generate executable Ansible playbooks directly from diagnosis reports to recover real microservice systems. It couples a dynamic failure-injection pipeline with three systems and nine LLMs, and proposes ThinkRemed, a four-agent, SRE-inspired framework that iteratively probes, reasons, executes, and verifies remediation actions. Experimental results show substantial challenges for current LLMs, with ThinkRemed delivering about a 7% accuracy improvement over one-shot baselines, especially on complex failure types, albeit at higher latency and token costs. The work demonstrates the potential and practicality of automated, LLM-driven microservice remediation while outlining limitations and avenues for refinement, such as adaptive probing and more efficient reflection strategies.

Abstract

Large Language Models (LLMs) integrated with agent-based reasoning frameworks have recently shown strong potential for autonomous decision-making and system-level operations. One promising yet underexplored direction is microservice remediation, where the goal is to automatically recover faulty microservice systems. Existing approaches, however, still rely on human-crafted prompts from Site Reliability Engineers (SREs), with LLMs merely converting textual instructions into executable code. To advance research in this area, we introduce MicroRemed, the first benchmark for evaluating LLMs in end-to-end microservice remediation, where models must directly generate executable Ansible playbooks from diagnosis reports to restore system functionality. We further propose ThinkRemed, a multi-agent framework that emulates the reflective and perceptive reasoning of SREs. Experimental results show that MicroRemed presents substantial challenges to current LLMs, while ThinkRemed improves end-to-end remediation performance through iterative reasoning and system reflection. The benchmark is available at https://github.com/LLM4AIOps/MicroRemed.

Figures (13)

• Figure 1: Previous microservice remediation workflow compared with the end-to-end microservice remediation pipeline proposed in MicroRemed.
• Figure 2: MicroRemed Benchmark Pipeline: the benchmark launches a real microservice; Failure Injection injects faults and produces a Failure Report; the Failure Report together with Auxiliary Context is provided to the Candidate Remediation LLM which generates an Ansible Playbook; the Execution Engine executes the playbook; Status Verification checks remediation success; Evaluation and Recovery restores the system for the next run.
• Figure 3: The overall framework of ThinkRemed
• Figure 4: Latency–accuracy trade-off of various large language models across three difficulty levels (Easy, Medium, Hard) on the Online-Boutique microservice
• Figure 5: Class-wise performance comparison across failure types on the Train-Ticket and Online-Boutique