Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions

Peter Barnett, Aaron Scher

TL;DR

The paper argues that humanity faces existential risks from rapidly advancing AI as frontier systems approach superintelligence. It proposes a structured landscape of four governance trajectories—Off Switch/Halt, US National Project, Light-Touch, and Threat of Sabotage—and catalogs hundreds of governance questions to guide research and policy. The central thesis is that building an international Off Switch and a credible Halt offers the best path to reduce extinction risk, with the US National Project deemed dangerous and not endorsed. The work emphasizes urgent, cross-sector action to develop monitoring, verification, and enforcement mechanisms, and to prepare for international agreements that can sustainably restrict dangerous AI activities. Overall, it provides a pragmatic, risk-focused roadmap for governance research intended to buy time for safe AI development and global cooperation.

Abstract

Humanity appears to be on course to soon develop AI systems that substantially outperform human experts in all cognitive domains and activities. We believe the default trajectory has a high likelihood of catastrophe, including human extinction. Risks come from failure to control powerful AI systems, misuse of AI by malicious rogue actors, war between great powers, and authoritarian lock-in. This research agenda has two aims: to describe the strategic landscape of AI development and to catalog important governance research questions. These questions, if answered, would provide important insight on how to successfully reduce catastrophic risks. We describe four high-level scenarios for the geopolitical response to advanced AI development, cataloging the research questions most relevant to each. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. The second scenario we describe is a US National Project for AI, in which the US Government races to develop advanced AI systems and establish unilateral control over global AI development. We also describe two additional scenarios: a Light-Touch world similar to that of today and a Threat of Sabotage situation where countries use sabotage and deterrence to slow AI development. In our view, apart from the Off Switch and Halt scenario, all of these trajectories appear to carry an unacceptable risk of catastrophic harm. Urgent action is needed from the US National Security community and AI governance ecosystem to answer key research questions, build the capability to halt dangerous AI activities, and prepare for international AI agreements.

AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions

TL;DR

The paper argues that humanity faces existential risks from rapidly advancing AI as frontier systems approach superintelligence. It proposes a structured landscape of four governance trajectories—Off Switch/Halt, US National Project, Light-Touch, and Threat of Sabotage—and catalogs hundreds of governance questions to guide research and policy. The central thesis is that building an international Off Switch and a credible Halt offers the best path to reduce extinction risk, with the US National Project deemed dangerous and not endorsed. The work emphasizes urgent, cross-sector action to develop monitoring, verification, and enforcement mechanisms, and to prepare for international agreements that can sustainably restrict dangerous AI activities. Overall, it provides a pragmatic, risk-focused roadmap for governance research intended to buy time for safe AI development and global cooperation.

Abstract

Humanity appears to be on course to soon develop AI systems that substantially outperform human experts in all cognitive domains and activities. We believe the default trajectory has a high likelihood of catastrophe, including human extinction. Risks come from failure to control powerful AI systems, misuse of AI by malicious rogue actors, war between great powers, and authoritarian lock-in. This research agenda has two aims: to describe the strategic landscape of AI development and to catalog important governance research questions. These questions, if answered, would provide important insight on how to successfully reduce catastrophic risks. We describe four high-level scenarios for the geopolitical response to advanced AI development, cataloging the research questions most relevant to each. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. The second scenario we describe is a US National Project for AI, in which the US Government races to develop advanced AI systems and establish unilateral control over global AI development. We also describe two additional scenarios: a Light-Touch world similar to that of today and a Threat of Sabotage situation where countries use sabotage and deterrence to slow AI development. In our view, apart from the Off Switch and Halt scenario, all of these trajectories appear to carry an unacceptable risk of catastrophic harm. Urgent action is needed from the US National Security community and AI governance ecosystem to answer key research questions, build the capability to halt dangerous AI activities, and prepare for international AI agreements.
Paper Structure (54 sections, 2 figures, 4 tables)

This paper contains 54 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Who we are
  3. About this document
  4. Core considerations
  5. Core risks to avoid
  6. Factors that worsen these risks
  7. Factors that reduce these risks
  8. Other background claims
  9. Halting dangerous AI development
  10. High-level scenarios for AI development
  11. Research project taxonomy
  12. Off Switch and Halt
  13. Off Switch
  14. Acute risk mitigation
  15. Short-term pause
  16. ...and 39 more sections

Figures (2)

  • Figure 1: A rough tree of the scenarios discussed in this report and how AI governance paradigms may evolve. This diagram assumes catastrophe is avoided at each step, so the myriad failures are omitted. This is simplified, and one could draw many more connections between the scenarios.
  • Figure 2: A rough tree of how AI governance paradigms may evolve, assuming catastrophe is avoided at each step. This is simplified, and one could draw many more connections between the scenarios. This figure is duplicated in the Executive Summary (Figure \ref{['fig:paradigm_tree_exec_summary']}).