What are the odds? Risk and uncertainty about AI existential risk

TL;DR

The paper analyzes risk and uncertainty in AI existential risk using a four-layer Swiss Cheese framework (Technical Plateau, Cultural Plateau, Alignment, Oversight) to compute $P(D)$ and critiques the linear, independence-assuming nature of such models. It shows that under epistemic indifference the doom probability can rise from $P(D) = (0.5)^4 = 6.25\%$ to around $10.416\%$ when refining layer structure (e.g., splitting Oversight into multiple submethods), illustrating how structure and partial knowledge shape risk. It introduces State-Space Uncertainty (SU) and Option Uncertainty (OU) as two key sources of unmodeled risk: SU concerns unknown survival stories and their interactions (no-calibration and unknown-relations problems), while OU captures non-binary, reflexive dynamics that can alter risk through feedbacks (e.g., Cultural Plateau affecting Oversight and Alignment). The contribution is a principled reminder to incorporate uncertainty and inter-layer structure into risk assessment, guiding more robust safety strategies for AI development.

Abstract

This work is a commentary of the article \href{https://doi.org/10.18716/ojs/phai/2025.2801}{AI Survival Stories: a Taxonomic Analysis of AI Existential Risk} by Cappelen, Goldstein, and Hawthorne. It is not just a commentary though, but a useful reminder of the philosophical limitations of \say{linear} models of risk. The article will focus on the model employed by the authors: first, I discuss some differences between standard Swiss Cheese models and this one. I then argue that in a situation of epistemic indifference the probability of P(D) is higher than what one might first suggest, given the structural relationships between layers. I then distinguish between risk and uncertainty, and argue that any estimation of P(D) is structurally affected by two kinds of uncertainty: option uncertainty and state-space uncertainty. Incorporating these dimensions of uncertainty into our qualitative discussion on AI existential risk can provide a better understanding of the likeliness of P(D).