FAIL: Analyzing Software Failures from the News Using LLMs

TL;DR

FAIL introduces an automated pipeline that uses LLMs to collect, cluster, and analyze software failures described in the news, enabling large-scale failure-aware software development. The system achieves high automation with $F_1=0.90$ and $V$-measure=0.98 on core tasks, processing 137,427 articles to identify 2,457 distinct failures across 4,184 articles from 2010–2022. The resulting FAIL database supports insights into recurring failure patterns, increasing severity of consequences, and cybersecurity implications, and is publicly available for researchers, engineers, and policymakers. This work demonstrates that modern LLMs can perform structured postmortem extraction from publicly available news data, offering a scalable resource for policy, engineering, and academic applications. It also outlines extensions such as thematic analyses and a failure-aware chatbot to integrate failure knowledge into the software development lifecycle.

Abstract

Software failures inform engineering work, standards, regulations. For example, the Log4J vulnerability brought government and industry attention to evaluating and securing software supply chains. Accessing private engineering records is difficult, so failure analyses tend to use information reported by the news media. However, prior works in this direction have relied on manual analysis. That has limited the scale of their analyses. The community lacks automated support to enable such analyses to consider a wide range of news sources and incidents. In this paper, we propose the Failure Analysis Investigation with LLMs (FAIL) system to fill this gap. FAIL collects, analyzes, and summarizes software failures as reported in the news. FAIL groups articles that describe the same incidents. It then analyzes incidents using existing taxonomies for postmortems, faults, and system characteristics. To tune and evaluate FAIL, we followed the methods of prior works by manually analyzing 31 software failures. FAIL achieved an F1 score of 90% for collecting news about software failures, a V-measure of 0.98 for merging articles reporting on the same incident, and extracted 90% of the facts about failures. We then applied FAIL to a total of 137,427 news articles from 11 providers published between 2010 and 2022. FAIL identified and analyzed 2457 distinct failures reported across 4,184 articles. Our findings include: (1) current generation of large language models are capable of identifying news articles that describe failures, and analyzing them according to structured taxonomies; (2) high recurrences of similar failures within organizations and across organizations; and (3) severity of the consequences of software failures have increased over the past decade. The full FAIL database is available so that researchers, engineers, and policymakers can learn from a diversity of software failures.

Figures (6)

• Figure 1: Concept of the FAIL (Failure Analysis Investigation with LLMs) project. Software failures are often described in news articles, e.g., the examples here for the Log4j security vulnerability. If analysts can identify and collate related articles about failures, they can share this knowledge with software engineers and policymakers. Current approaches are manual. We propose the FAIL system to automatically collect, group, and analyze software failures from the news. Our FAIL prototype collected, grouped, and analyzed these articles together for Log4j.
• Figure 2: Overview of the proposed FAIL pipeline to collect, analyze, and report insights from software failures reported in the news. FAIL has 6 components. In Step 1, we search for news based on criteria such as keywords, sources, and timeframe. In Step 2, we remove articles that are not about software failures. In Step 3, we remove articles that do not have enough information to analyze. In Step 4, we merge articles reporting on the same incident. In Step 5, we use RAG techniques to handle long incidents. In Step 6, we create a failure report for each incident.
• Figure 3: (a) Incidents by keyword that found them in Step 1. (b) Distribution of news sources.
• Figure 4: Frequency of articles (x-axis) to incidents (y-axis, log scale). The most-covered incidents were the Boeing 737 MAX crashes, Sony PlayStation network attacks, and Tesla Autopilot failures.
• Figure 5: Taxonimization of the characteristics of the causes of recent software failures from our database.