Legally-Informed Explainable AI

TL;DR

This paper argues for Legally-Informed Explainable AI by integrating legal considerations into AI explanations used in high-stakes domains. It develops a stakeholder-centered framework focusing on decision makers, decision subjects, and legal representatives, and illustrates the approach with a healthcare case study highlighting evolving technology-law interactions. The authors propose concrete design opportunities—legally informative content for clinicians and legally-informed evaluations—and advocate co-disciplinary analyses of litigation and harms to guide development. The goal is to preserve agency, enable contestation, and ensure explanations translate into lawful and actionable guidance for all stakeholders. The work emphasizes infrastructure for ongoing, multi-stakeholder collaboration to adapt explanations to a dynamic legal landscape.

Abstract

Explanations for artificial intelligence (AI) systems are intended to support the people who are impacted by AI systems in high-stakes decision-making environments, such as doctors, patients, teachers, students, housing applicants, and many others. To protect people and support the responsible development of AI, explanations need to be actionable--helping people take pragmatic action in response to an AI system--and contestable--enabling people to push back against an AI system and its determinations. For many high-stakes domains, such as healthcare, education, and finance, the sociotechnical environment includes significant legal implications that impact how people use AI explanations. For example, physicians who use AI decision support systems may need information on how accepting or rejecting an AI determination will protect them from lawsuits or help them advocate for their patients. In this paper, we make the case for Legally-Informed Explainable AI, responding to the need to integrate and design for legal considerations when creating AI explanations. We describe three stakeholder groups with different informational and actionability needs, and provide practical recommendations to tackle design challenges around the design of explainable AI systems that incorporate legal considerations.