Two Types of AI Existential Risk: Decisive and Accumulative
Atoosa Kasirzadeh
TL;DR
The paper critiques the prevailing focus on decisive AI x-risk by proposing an accumulative AI x-risk framework in which numerous smaller, interconnected AI-driven disruptions gradually erode societal resilience, culminating in an existential collapse. It employs systems analysis to compare the decisive and accumulative pathways, introduces the perfect storm MISTER scenario to illustrate how social and technical risks interact, and argues that governance must address both pathways with a tiered, holistic approach. Key contributions include reframing AI risk as a networked, time-dependent process, articulating how ethical and social risks can compound into existential threats, and outlining concrete governance implications such as distributed monitoring and cross-domain risk unification. The work highlights the practical significance of monitoring, simulation, and formal modeling to detect early warning signals and prevent irreversible systemic failure in a highly interconnected AI-enabled world.
Abstract
The conventional discourse on existential risks (x-risks) from AI typically focuses on abrupt, dire events caused by advanced AI systems, particularly those that might achieve or surpass human-level intelligence. These events have severe consequences that either lead to human extinction or irreversibly cripple human civilization to a point beyond recovery. This discourse, however, often neglects the serious possibility of AI x-risks manifesting incrementally through a series of smaller yet interconnected disruptions, gradually crossing critical thresholds over time. This paper contrasts the conventional "decisive AI x-risk hypothesis" with an "accumulative AI x-risk hypothesis." While the former envisions an overt AI takeover pathway, characterized by scenarios like uncontrollable superintelligence, the latter suggests a different causal pathway to existential catastrophes. This involves a gradual accumulation of critical AI-induced threats such as severe vulnerabilities and systemic erosion of economic and political structures. The accumulative hypothesis suggests a boiling frog scenario where incremental AI risks slowly converge, undermining societal resilience until a triggering event results in irreversible collapse. Through systems analysis, this paper examines the distinct assumptions differentiating these two hypotheses. It is then argued that the accumulative view can reconcile seemingly incompatible perspectives on AI risks. The implications of differentiating between these causal pathways -- the decisive and the accumulative -- for the governance of AI as well as long-term AI safety are discussed.