The Role of Risk Modeling in Advanced AI Risk Management

TL;DR

The paper argues that mature AI risk management requires risk modeling that tightly couples scenario building with quantitative risk estimation to address epistemic uncertainty. It operationalizes AI risk modeling, adapts established tools (FTA, ETA, FMEA/FMECA, STPA, bow-tie) and quantitative methods (Monte Carlo, Bayesian networks, copulas, structured expert elicitation) to AI contexts, and emphasizes dependency-aware, iterative modeling. By surveying five safety-critical sectors—nuclear, aviation, cybersecurity, finance, and submarines—it derives governance lessons and advocates for a hybrid deterministic-probabilistic risk framework plus verifiable AI safety components. The authors propose a governance-ready framing linking model outputs to societal risk tolerance and call for continued research in provably safe AI architectures to deliver stronger, more transparent risk management for frontier AI.

Abstract

Rapidly advancing artificial intelligence (AI) systems introduce novel, uncertain, and potentially catastrophic risks. Managing these risks requires a mature risk-management infrastructure whose cornerstone is rigorous risk modeling. We conceptualize AI risk modeling as the tight integration of (i) scenario building$-$causal mapping from hazards to harms$-$and (ii) risk estimation$-$quantifying the likelihood and severity of each pathway. We review classical techniques such as Fault and Event Tree Analyses, FMEA/FMECA, STPA and Bayesian networks, and show how they can be adapted to advanced AI. A survey of emerging academic and industry efforts reveals fragmentation: capability benchmarks, safety cases, and partial quantitative studies are valuable but insufficient when divorced from comprehensive causal scenarios. Comparing the nuclear, aviation, cybersecurity, financial, and submarine domains, we observe that every sector combines deterministic guarantees for unacceptable events with probabilistic assessments of the broader risk landscape. We argue that advanced-AI governance should adopt a similar dual approach and that verifiable, provably-safe AI architectures are urgently needed to supply deterministic evidence where current models are the result of opaque end-to-end optimization procedures rather than specified by hand. In one potential governance-ready framework, developers conduct iterative risk modeling and regulators compare the results with predefined societal risk tolerance thresholds. The paper provides both a methodological blueprint and opens a discussion on the best way to embed sound risk modeling at the heart of advanced-AI risk management.

Figures (4)

• Figure 1: Excerpt from the risk matrix in the UK National Risk Register 2025. Source: UK National Risk Register, UKNRR2025. Please note that the matrix reproduced here is truncated due to space constraints; some of the numbers in the blue part are missing from the list on the right.
• Figure 2: A simple Bayesian Network illustrating relationships between AI misalignment, insufficient testing, adversarial attacks, and harmful outcomes Source: authors' own production.
• Figure 3: A potential framework for integrating risk modeling into risk management campos2025frontier
• Figure :