Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations

TL;DR

The paper addresses the safe integration of AI in medical devices by foregrounding explainability. It employs a UK MHRA–led expert group and a two-stage study (workshops and a clinician pilot) to compare simple and complex AI approaches for heart-attack risk prediction, using LIME, odds-ratio/global explanations, GINI-based measures, and ExMatrix counterfactuals. Key findings indicate that regulators favor interpretability and local explanations, while clinicians value performance and regulatory alignment; explanations can boost clinician confidence but may also cause automation bias if misaligned. The work yields practical recommendations for regulatory assessment, model selection, and stakeholder training to safely implement AI-driven CDSS in clinical practice.

Abstract

There is a growing demand for the use of Artificial Intelligence (AI) and Machine Learning (ML) in healthcare, particularly as clinical decision support systems to assist medical professionals. However, the complexity of many of these models, often referred to as black box models, raises concerns about their safe integration into clinical settings as it is difficult to understand how they arrived at their predictions. This paper discusses insights and recommendations derived from an expert working group convened by the UK Medicine and Healthcare products Regulatory Agency (MHRA). The group consisted of healthcare professionals, regulators, and data scientists, with a primary focus on evaluating the outputs from different AI algorithms in clinical decision-making contexts. Additionally, the group evaluated findings from a pilot study investigating clinicians' behaviour and interaction with AI methods during clinical diagnosis. Incorporating AI methods is crucial for ensuring the safety and trustworthiness of medical AI devices in clinical settings. Adequate training for stakeholders is essential to address potential issues, and further insights and recommendations for safely adopting AI systems in healthcare settings are provided.

Figures (2)

• Figure 1: Tradeoff between Performance and Complexity of the model Vs Perceived Interpretabilty in Machine Learning
• Figure 6: Clinical Confidence Vs ANN model Confidence in Diagnosis Patients as High Risk to Have a Heart Attack