Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systems
Robin Bloomfield, John Rushby
TL;DR
The paper argues that AI assurance can fail by targeting the wrong risks, employing inadequate methods, or failing to communicate the rationale to stakeholders. It draws on traditional critical-systems engineering to map AI safety into environment-aware system definitions, hazard and risk analysis, and rigorous assurance cases, emphasizing defence-in-depth and broad system boundaries. A central claim is that Assurance 2.0—with structured arguments, defeaters, and explicit confidence accounting—offers a skeptical, theory-grounded path to justify safety claims in AI, especially for high-stakes deployments and Foundation Models. The authors call for resilience, open, interdisciplinary dialogue, and adaptation of best practices from established industries to ensure AI safety frameworks scale fundamentally beyond everyday systems.
Abstract
We draw on our experience working on system and software assurance and evaluation for systems important to society to summarise how safety engineering is performed in traditional critical systems, such as aircraft flight control. We analyse how this critical systems perspective might support the development and implementation of AI Safety Frameworks. We present the analysis in terms of: system engineering, safety and risk analysis, and decision analysis and support. We consider four key questions: What is the system? How good does it have to be? What is the impact of criticality on system development? and How much should we trust it? We identify topics worthy of further discussion. In particular, we are concerned that system boundaries are not broad enough, that the tolerability and nature of the risks are not sufficiently elaborated, and that the assurance methods lack theories that would allow behaviours to be adequately assured. We advocate the use of assurance cases based on Assurance 2.0 to support decision making in which the criticality of the decision as well as the criticality of the system are evaluated. We point out the orders of magnitude difference in confidence needed in critical rather than everyday systems and how everyday techniques do not scale in rigour. Finally we map our findings in detail to two of the questions posed by the FAISC organisers and we note that the engineering of critical systems has evolved through open and diverse discussion. We hope that topics identified here will support the post-FAISC dialogues.