The Start Button Problem: a basis for human responsibility in artificial intelligence computation

TL;DR

This paper interrogates whether artificial general intelligence can exhibit genuine agency by introducing the Start Button Problem, a thought experiment that centers on human-initiated activation as a prerequisite for AI operation. It analyzes definitions of AGI and draws on the ARC GPT-4 experiment to illustrate how generality can emerge alongside constrained agency, but argues that true agency remains contingent on external initiation. The authors conclude that, under current architectures, AI remains dependent on human input, which preserves responsibility with the initiator while complicating, but does not erase, moral blame for long-term, unintended outcomes. The work has implications for how we conceptualize AGI, the responsibility gap, and the design of AI systems, highlighting the ongoing tension between automation, delegation, emergent behaviour, and human accountability.

Abstract

Recent advancements in artificial intelligence have reopened the question about the boundaries of AI autonomy, particularly in discussions around artificial general intelligence and its potential to act independently across varied purposes. This paper explores these boundaries through the analysis of the Alignment Research Center experiment on GPT-4 and introduces the Start Button Problem, a thought experiment that examines the origins and limits of AI autonomy. By examining the thought experiment and its counterarguments, it becomes clear that in its need for human activation and purpose definition lies the AI's inherent dependency on human-initiated actions, challenging the assumption of AI as an intelligent agent. Finally, the paper addresses the implications of this dependency on human responsibility, questioning the measure of the extension of human responsibility when using AI systems.