Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The EU AI Act, Stakeholder Needs, and Explainable AI: Aligning Regulatory Compliance in a Clinical Decision Support System

Anton Hummel, Håkan Burden, Susanne Stenberg, Jan-Philipp Steghöfer, Niklas Kühl

TL;DR

The paper tackles the mismatch between the EU AI Act's provider/deployer focus and end-user needs in high-risk healthcare, proposing a cross-disciplinary framework to align regulatory compliance with Explainable AI practices in a CDSS. Using a real-world ICU CDSS scenario, the authors perform parallel legal and XAI analyses and merge them to map AI Act obligations to stakeholder desiderata and to XAI techniques. Key contributions include a practical methodology for stakeholder-driven XAI selection, a CE-marking and conformity pathway assessment for a CDSS, and actionable recommendations for integration into product development and governance. The work advances policy-relevant standards development by illustrating how XAI can support transparency and human oversight within a broader compliance program.

Abstract

Explainable AI (XAI) is a promising route to comply with the EU AI Act, the first multinational AI regulation. XAI enhances transparency and human oversight of AI systems, especially ''black-box`` models criticized as incomprehensible. Yet discourse about the AI Act's stakeholders and XAI remains disconnected: XAI increasingly prioritizes end users' needs, while the AI Act focuses on providers' and deployers' obligations. We aim to bridge this divide and offer practical guidance on their relationship. Through interdisciplinary discussion in a cross functional team of XAI, AI Act, legal, and requirements-engineering experts, we outline steps to analyze an AI-based clinical decision support system, clarify end-user needs, and assess AI Act applicability. Using an AI system under development as a case study, we show how XAI techniques can help reconcile stakeholder needs with AI Act requirements and fill gaps between usability and regulatory demands. We compare similarities and differences between legal obligations and end-user needs, identify tensions, and point to concrete design choices and trade-offs. We invite researchers and practitioners in XAI to reflect on their role relative to the AI Act and to develop mutual understanding across disciplines. While XAI can help implement core AI Act principles such as transparency and human oversight, it should be considered one element of a broader compliance strategy that also requires standardization, legal interpretation, documentation, organizational processes, governance, testing, and ongoing monitoring and auditing practices. Our findings yield actionable recommendations for integrating XAI into product development, compliance workflows, and stakeholder communication, informing policy-making and standards development.

The EU AI Act, Stakeholder Needs, and Explainable AI: Aligning Regulatory Compliance in a Clinical Decision Support System

TL;DR

The paper tackles the mismatch between the EU AI Act's provider/deployer focus and end-user needs in high-risk healthcare, proposing a cross-disciplinary framework to align regulatory compliance with Explainable AI practices in a CDSS. Using a real-world ICU CDSS scenario, the authors perform parallel legal and XAI analyses and merge them to map AI Act obligations to stakeholder desiderata and to XAI techniques. Key contributions include a practical methodology for stakeholder-driven XAI selection, a CE-marking and conformity pathway assessment for a CDSS, and actionable recommendations for integration into product development and governance. The work advances policy-relevant standards development by illustrating how XAI can support transparency and human oversight within a broader compliance program.

Abstract

Explainable AI (XAI) is a promising route to comply with the EU AI Act, the first multinational AI regulation. XAI enhances transparency and human oversight of AI systems, especially ''black-box`` models criticized as incomprehensible. Yet discourse about the AI Act's stakeholders and XAI remains disconnected: XAI increasingly prioritizes end users' needs, while the AI Act focuses on providers' and deployers' obligations. We aim to bridge this divide and offer practical guidance on their relationship. Through interdisciplinary discussion in a cross functional team of XAI, AI Act, legal, and requirements-engineering experts, we outline steps to analyze an AI-based clinical decision support system, clarify end-user needs, and assess AI Act applicability. Using an AI system under development as a case study, we show how XAI techniques can help reconcile stakeholder needs with AI Act requirements and fill gaps between usability and regulatory demands. We compare similarities and differences between legal obligations and end-user needs, identify tensions, and point to concrete design choices and trade-offs. We invite researchers and practitioners in XAI to reflect on their role relative to the AI Act and to develop mutual understanding across disciplines. While XAI can help implement core AI Act principles such as transparency and human oversight, it should be considered one element of a broader compliance strategy that also requires standardization, legal interpretation, documentation, organizational processes, governance, testing, and ongoing monitoring and auditing practices. Our findings yield actionable recommendations for integrating XAI into product development, compliance workflows, and stakeholder communication, informing policy-making and standards development.

Paper Structure

This paper contains 29 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work and Background
  3. Background: The EU AI Act
  4. CE-marking AI Systems
  5. Obligations of the Provider
  6. Obligations of the Deployer
  7. Background: XAI Terminology
  8. Explainability, Interpretability, and Transparency
  9. Stakeholder Terminology
  10. Taxonomy of Explainability Methods
  11. Related Work
  12. Methodology
  13. The Cross-functional Team
  14. The Clinical Decision Support System
  15. Legal Analysis
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Example Heatmap Explanation with Shap. Shows the five most important features in descending order of relevance over a period of 14 hours. The color gradient indicates whether the feature value at a given timestamp has a positive (red) or negative (positive) impact on the predicted los.
  • Figure 2: Stakeholder roles and their responsibilities in XAI and the AI Act. Italics denote XAI roles, bold denotes AI Act roles. Parenthesized roles are from the cdss. An arrow from A to B denotes that A has a responsibility towards B.