Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds

TL;DR

The paper argues that AI risk thresholds lack democratically determined baselines, risking a shift toward safety revisionism akin to an autocratic value system. It traces this through Chauncey Starr’s probabilistic risk analysis and contrasts nuclear-era risk governance with today’s AI discourse, criticizing the conflation of safety with alignment and capabilities. The authors show how safety cases and generic benchmarks are co-opted to justify accelerated deployment in defense contexts, often at odds with TEVV standards and IHL. They advocate restoring risk-threshold deliberation, preserving TEVV-based evaluation for foundation-model defense applications, and creating democratic governance mechanisms to set concrete, context-specific safety goals that protect both national security and civilian safety.

Abstract

Risk thresholds provide a measure of the level of risk exposure that a society or individual is willing to withstand, ultimately shaping how we determine the safety of technological systems. Against the backdrop of the Cold War, the first risk analyses, such as those devised for nuclear systems, cemented societally accepted risk thresholds against which safety-critical and defense systems are now evaluated. But today, the appropriate risk tolerances for AI systems have yet to be agreed on by global governing efforts, despite the need for democratic deliberation regarding the acceptable levels of harm to human life. Absent such AI risk thresholds, AI technologists-primarily industry labs, as well as "AI safety" focused organizations-have instead advocated for risk tolerances skewed by a purported AI arms race and speculative "existential" risks, taking over the arbitration of risk determinations with life-or-death consequences, subverting democratic processes. In this paper, we demonstrate how such approaches have allowed AI technologists to engage in "safety revisionism," substituting traditional safety methods and terminology with ill-defined alternatives that vie for the accelerated adoption of military AI uses at the cost of lowered safety and security thresholds. We explore how the current trajectory for AI risk determination and evaluation for foundation model use within national security is poised for a race to the bottom, to the detriment of the US's national security interests. Safety-critical and defense systems must comply with assurance frameworks that are aligned with established risk thresholds, and foundation models are no exception. As such, development of evaluation frameworks for AI-based military systems must preserve the safety and security of US critical and defense infrastructure, and remain in alignment with international humanitarian law.