Legitimate Power, Illegitimate Automation: The problem of ignoring legitimacy in automated decision systems

TL;DR

The paper tackles the legitimacy of automated decision systems (ADS), arguing that existing focus on fairness, accuracy, and efficiency fails to address whether those in control of ADS are entitled to wield power. It critiques descriptive legitimacy for resting on public belief rather than normative grounds and examines three normative approaches—consent, public reason, and democratic authorisation—as potential foundations for ADS legitimacy. The authors contend that consent is often infeasible in high-stakes ADS contexts due to opacity and involuntariness, that public reason is valuable but frequently incomplete for concrete design questions, and that democratic authorisation offers a promising, though imperfect, route requiring regulatory oversight and expert input. The paper ultimately calls for an interdisciplinary research program to tailor legitimation strategies to specific ADS contexts, balancing equal influence with practical feasibility and social equity.

Abstract

Progress in machine learning and artificial intelligence has spurred the widespread adoption of automated decision systems (ADS). An extensive literature explores what conditions must be met for these systems' decisions to be fair. However, questions of legitimacy -- why those in control of ADS are entitled to make such decisions -- have received comparatively little attention. This paper shows that when such questions are raised theorists often incorrectly conflate legitimacy with either public acceptance or other substantive values such as fairness, accuracy, expertise or efficiency. In search of better theories, we conduct a critical analysis of the philosophical literature on the legitimacy of the state, focusing on consent, public reason, and democratic authorisation. This analysis reveals that the prevailing understanding of legitimacy in analytical political philosophy is also ill-suited to the task of establishing whether and when ADS are legitimate. The paper thus clarifies expectations for theories of ADS legitimacy and charts a path for a future research programme on the topic.