Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis
Evelien Riddell, James Riddell, Gengyi Sun, Michał Antkiewicz, Krzysztof Czarnecki
TL;DR
This work presents a reasoning-centered framework to evaluate LLMs for cloud-based RCA by isolating purely the model's reasoning from confounding multi-agent or system-specific factors. Using two real-world RCA benchmarks (GAIA and OpenRCA), a typed knowledge graph, and three agentic workflows (ReAct, Plan-and-Execute, Straight-Shot), the authors conduct a large-scale study (48k scenarios) and systematically annotate reasoning failures with an LLM-as-a-Judge. They introduce a 16-item reasoning-failure taxonomy spanning procedural, RCA-specific, and general categories, and show that reasoning failures are pervasive and strongly predictive of incorrect hypotheses, with metrics and certain modalities contributing differently to performance. The findings stress that agentic structure alone is not a silver bullet for RCA and highlight the need for explicit domain guidance, hypothesis diversification, and checks for evidence sufficiency to improve reasoning-driven system diagnosis in practice.
Abstract
Root cause analysis (RCA) is essential for diagnosing failures within complex software systems to ensure system reliability. The highly distributed and interdependent nature of modern cloud-based systems often complicates RCA efforts, particularly for multi-hop fault propagation, where symptoms appear far from their true causes. Recent advancements in Large Language Models (LLMs) present new opportunities to enhance automated RCA. However, their practical value for RCA depends on the fidelity of reasoning and decision-making. Existing work relies on historical incident corpora, operates directly on high-volume telemetry beyond current LLM capacity, or embeds reasoning inside complex multi-agent pipelines -- conditions that obscure whether failures arise from reasoning itself or from peripheral design choices. We present a focused empirical evaluation that isolates an LLM's reasoning behavior. We design a controlled experimental framework that foregrounds the LLM by using a simplified experimental setting. We evaluate six LLMs under two agentic workflows (ReAct and Plan-and-Execute) and a non-agentic baseline on two real-world case studies (GAIA and OpenRCA). In total, we executed 48,000 simulated failure scenarios, totaling 228 days of execution time. We measure both root-cause accuracy and the quality of intermediate reasoning traces. We produce a labeled taxonomy of 16 common RCA reasoning failures and use an LLM-as-a-Judge for annotation. Our results clarify where current open-source LLMs succeed and fail in multi-hop RCA, quantify sensitivity to input data modalities, and identify reasoning failures that predict final correctness. Together, these contributions provide transparent and reproducible empirical results and a failure taxonomy to guide future work on reasoning-driven system diagnosis.