Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Microservices: Testing Web-Scale RCA Methods on GPU-Driven LLM Workloads

Dominik Scheinert, Alexander Acker, Thorsten Wittkopp, Soeren Becker, Hamza Yous, Karnakar Reddy, Ibrahim Farhat, Hakim Hacid, Odej Kao

TL;DR

Evaluating the effectiveness of RCA methods on a best-practice LLM inference deployment under controlled failure injections reveals that existing RCA tools do not generalize to LLM systems, motivating tailored analysis techniques and enhanced observability.

Abstract

Large language model (LLM) services have become an integral part of search, assistance, and decision-making applications. However, unlike traditional web or microservices, the hardware and software stack enabling LLM inference deployment is of higher complexity and far less field-tested, making it more susceptible to failures that are difficult to resolve. Keeping outage costs and quality of service degradations in check depends on shortening mean time to repair, which in practice is gated by how quickly the fault is identified, located, and diagnosed. Automated root cause analysis (RCA) accelerates failure localization by identifying the system component that failed and tracing how the failure propagated. Numerous RCA methods have been developed for traditional services, using request path tracing, resource metric and log data analysis. Yet, existing RCA methods have not been designed for LLM deployments that present distinct runtime characteristics. In this study, we evaluate the effectiveness of RCA methods on a best-practice LLM inference deployment under controlled failure injections. Across 24 methods (20 metric-based, two trace-based, and two multi-source), we find that multi-source approaches achieve the highest accuracy, metric-based methods show fault-type-dependent performance, and trace-based methods largely fail. These results reveal that existing RCA tools do not generalize to LLM systems, motivating tailored analysis techniques and enhanced observability, for which we formulate guidelines.

Beyond Microservices: Testing Web-Scale RCA Methods on GPU-Driven LLM Workloads

TL;DR

Evaluating the effectiveness of RCA methods on a best-practice LLM inference deployment under controlled failure injections reveals that existing RCA tools do not generalize to LLM systems, motivating tailored analysis techniques and enhanced observability.

Abstract

Large language model (LLM) services have become an integral part of search, assistance, and decision-making applications. However, unlike traditional web or microservices, the hardware and software stack enabling LLM inference deployment is of higher complexity and far less field-tested, making it more susceptible to failures that are difficult to resolve. Keeping outage costs and quality of service degradations in check depends on shortening mean time to repair, which in practice is gated by how quickly the fault is identified, located, and diagnosed. Automated root cause analysis (RCA) accelerates failure localization by identifying the system component that failed and tracing how the failure propagated. Numerous RCA methods have been developed for traditional services, using request path tracing, resource metric and log data analysis. Yet, existing RCA methods have not been designed for LLM deployments that present distinct runtime characteristics. In this study, we evaluate the effectiveness of RCA methods on a best-practice LLM inference deployment under controlled failure injections. Across 24 methods (20 metric-based, two trace-based, and two multi-source), we find that multi-source approaches achieve the highest accuracy, metric-based methods show fault-type-dependent performance, and trace-based methods largely fail. These results reveal that existing RCA tools do not generalize to LLM systems, motivating tailored analysis techniques and enhanced observability, for which we formulate guidelines.
Paper Structure (25 sections, 2 equations, 8 figures, 1 table)

This paper contains 25 sections, 2 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. LLM Inference in Microservice Architectures
  3. Layered Architecture of LLM Inference Services
  4. Requirements for Observability
  5. RCA Methodology for LLM Inference
  6. Research Objective
  7. Enabling RCA for LLM Inference Architectures
  8. LLM Deployment Reference Stack
  9. Observability for LLM Inference Deployments
  10. Load Generation and Failure Injections
  11. Experiment Results
  12. Evaluation Metrics
  13. Performance Analysis of RCA Methods
  14. Metric-based Methods
  15. Trace-based Methods
  16. ...and 10 more sections

Figures (8)

  • Figure 1: Illustration of both traditional microservices and modern LLM inference services as part of a distributed service architecture. We highlight similarities and differences in their individual layered architectures.
  • Figure 2: High-level overview of our LLM inference deployment stack and observability tooling. The different architecture layers as well as telemetry data pillars, previously discussed in \ref{['sec:towards_rca_for_llm']}, are mapped to concrete technologies, eventually forming our experimental system and serving as a blueprint for other practitioners and researchers.
  • Figure 3: We present exemplary results of conducted smoke tests that show the number of processed requests over time when facing a constant load of queries per second (QPS) in light of different data parallelism (DP) configurations.
  • Figure 4: Abstract illustration of the setup and request handling workflow. A load generator sends request to the NGINX gateway that is exposed externally from the cluster. The gateway then forwards request to the Kubernetes service responsible for balancing serve requests for a particular LLM. Eventually, this service sends the request to one of the available endpoints, which includes a single Ray head Pod and any number of connected Ray worker Pods.
  • Figure 5: Misattribution counts of selected RCA methods across categories. Most approaches systematically assign faults to upstream components (NGINX Gateway or Ray head Pod) rather than the actual faulty Ray worker Pods.
  • ...and 3 more figures