Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models

Xiaoyu Chu, Sacheendra Talluri, Qingxian Lu, Alexandru Iosup

TL;DR

Public LLM services experience outages that impact usability and finances, motivating long-term, cross-service failure data collection. The paper assembles longitudinal outage and incident datasets for 8 services across 3 providers and performs a data-driven failure-recovery analysis, including mechanisms, timing, and availability. Key findings show that OpenAI services typically recover more slowly but fail less often than Anthropic's, failures exhibit clear weekly and monthly periodicity, and incidents co-occur more within the same provider, highlighting varying isolation properties. By releasing open, FAIR datasets and tooling, the work enables reproducibility and informs design choices for resilient, user-facing LLM deployments.

Abstract

People and businesses increasingly rely on public LLM services, such as ChatGPT, DALLE, and Claude. Understanding their outages, and particularly measuring their failure-recovery processes, is becoming a stringent problem. However, only limited studies exist in this emerging area. Addressing this problem, in this work we conduct an empirical characterization of outages and failure-recovery in public LLM services. We collect and prepare datasets for 8 commonly used LLM services across 3 major LLM providers, including market-leads OpenAI and Anthropic. We conduct a detailed analysis of failure recovery statistical properties, temporal patterns, co-occurrence, and the impact range of outage-causing incidents. We make over 10 observations, among which: (1) Failures in OpenAI's ChatGPT take longer to resolve but occur less frequently than those in Anthropic's Claude;(2) OpenAI and Anthropic service failures exhibit strong weekly and monthly periodicity; and (3) OpenAI services offer better failure-isolation than Anthropic services. Our research explains LLM failure characteristics and thus enables optimization in building and using LLM systems. FAIR data and code are publicly available on https://zenodo.org/records/14018219 and https://github.com/atlarge-research/llm-service-analysis.

An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models

TL;DR

Public LLM services experience outages that impact usability and finances, motivating long-term, cross-service failure data collection. The paper assembles longitudinal outage and incident datasets for 8 services across 3 providers and performs a data-driven failure-recovery analysis, including mechanisms, timing, and availability. Key findings show that OpenAI services typically recover more slowly but fail less often than Anthropic's, failures exhibit clear weekly and monthly periodicity, and incidents co-occur more within the same provider, highlighting varying isolation properties. By releasing open, FAIR datasets and tooling, the work enables reproducibility and informs design choices for resilient, user-facing LLM deployments.

Abstract

People and businesses increasingly rely on public LLM services, such as ChatGPT, DALLE, and Claude. Understanding their outages, and particularly measuring their failure-recovery processes, is becoming a stringent problem. However, only limited studies exist in this emerging area. Addressing this problem, in this work we conduct an empirical characterization of outages and failure-recovery in public LLM services. We collect and prepare datasets for 8 commonly used LLM services across 3 major LLM providers, including market-leads OpenAI and Anthropic. We conduct a detailed analysis of failure recovery statistical properties, temporal patterns, co-occurrence, and the impact range of outage-causing incidents. We make over 10 observations, among which: (1) Failures in OpenAI's ChatGPT take longer to resolve but occur less frequently than those in Anthropic's Claude;(2) OpenAI and Anthropic service failures exhibit strong weekly and monthly periodicity; and (3) OpenAI services offer better failure-isolation than Anthropic services. Our research explains LLM failure characteristics and thus enables optimization in building and using LLM systems. FAIR data and code are publicly available on https://zenodo.org/records/14018219 and https://github.com/atlarge-research/llm-service-analysis.
Paper Structure (18 sections, 4 equations, 11 figures, 8 tables)

This paper contains 18 sections, 4 equations, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Anatomy of an LLM-service Incident: Model and Example
  3. Model and Real-World Example
  4. LLM-Specific Terms and Metrics
  5. Dataset Collection and Preparation
  6. Selection and Introduction of LLM Services
  7. Data Collection and Dataset Preparation
  8. Failure-Recovery Analysis
  9. Failure Patterns Over Time
  10. Temporal Distributions
  11. Auto-correlations
  12. Service Availability Over Time
  13. Co-occurrence of Failures
  14. Co-occurrence of Outages
  15. Impact Range of Incidents
  16. ...and 3 more sections

Figures (11)

  • Figure 1: Monthly website visits, outages, and incidents for ChatGPT. Vertical axis: (left) number of website visits in billions; (right) monthly outage and incident counts. Data: website visits similarwab, outages (\ref{['tab:outage-dataset']}), and incidents (\ref{['tab:incident-dataset']}).
  • Figure 2: Visualization of the failure-recovery model with user reports of a selected ChatGPT incident, UDT time.
  • Figure 3: Presence of different status combinations, by service [%]. Due to small counts, status combinations with $S_5$ are merged into 'All-with-S5'. (\ref{['tab:outage-dataset']} indexes the services.)
  • Figure 4: Percent of time spent in the Investigating, Repairing, and Checking periods, from the overall duration for failure resolution [%].
  • Figure 5: Distribution of MTTR and MTBF by service, with median values indicated.
  • ...and 6 more figures