A quality of mercy is not trained: the imagined vs. the practiced in healthcare process-specialized AI development

TL;DR

Healthcare AI scheduling systems embed abstracted optimization that may overlook ethical discretion embedded in frontline coordination. The paper conducts an embedded qualitative study at a Canadian hospital to compare scheduling imagined by designers (rule-based, surgeon-centric) with scheduling practiced by clerks, nurses, and managers. It introduces the concept of epistemic foreclosure to describe how early representational choices narrow what the system can see, thereby silencing patient-centered and relational considerations. The authors argue for a situated ethics of AI that keeps design open to dialogic revision and responsive to shifting knowledges across roles, with implications for safer, more humane healthcare AI.

Abstract

In high stakes organizational contexts like healthcare, artificial intelligence (AI) systems are increasingly being designed to augment complex coordination tasks. This paper investigates how the ethical stakes of such systems are shaped by their epistemic framings: what aspects of work they represent, and what they exclude. Drawing on an embedded study of AI development for operating room (OR) scheduling at a Canadian hospital, we compare scheduling-as-imagined in the AI design process: rule-bound, predictable, and surgeon-centric, with scheduling-as-practiced as a fluid, patient-facing coordination process involving ethical discretion. We show how early representational decisions narrowed what the AI could support, resulting in epistemic foreclosure: the premature exclusion of key ethical dimensions from system design. Our findings surface the moral consequences of abstraction and call for a more situated approach to designing healthcare process-specialized artificial intelligence systems.