The Explanation Necessity for Healthcare AI

TL;DR

The paper tackles the lack of practical guidance for explainability in healthcare AI by proposing a four-class taxonomy of explanation necessity—self-explainable, semi-explainable, non-explainable, and new-patterns discovery—parameterized by three factors: the robustness of the evaluation protocol, variability in expert observations, and the representational dimensionality $d$. It provides a formal mathematical formulation with inputs $\mathbf{x}$, outputs $\mathbf{y}$, and post-hoc explanations $\mathbf{z}$, introducing global explanations $g_z=R(\mathbf{z})$ and an alignment mechanism $A_l$ for new-pattern discovery against a ground-truth model $R_S(\mathbf{x'})$. A two-flow framework guides practical decision-making: inter-observer variability assessment and dimensionality evaluation, culminating in an $XAI$ Need Decision and, when needed, a Category Decision. The approach aims to improve trust, safety, and regulatory readiness in clinical AI by linking explanation depth to task risk and data characteristics, with domain-specific examples illustrating how thresholds and classifications shift across medical imaging and diagnostic tasks. Future work is needed to normalize thresholds for other domains and to refine risk-adaptive explanations in high-stakes settings.

Abstract

Explainability is a critical factor in enhancing the trustworthiness and acceptance of artificial intelligence (AI) in healthcare, where decisions directly impact patient outcomes. Despite advancements in AI interpretability, clear guidelines on when and to what extent explanations are required in medical applications remain lacking. We propose a novel categorization system comprising four classes of explanation necessity (self-explainable, semi-explainable, non-explainable, and new-patterns discovery), guiding the required level of explanation; whether local (patient or sample level), global (cohort or dataset level), or both. To support this system, we introduce a mathematical formulation that incorporates three key factors: (i) robustness of the evaluation protocol, (ii) variability of expert observations, and (iii) representation dimensionality of the application. This framework provides a practical tool for researchers to determine the appropriate depth of explainability needed, addressing the critical question: When does an AI medical application need to be explained, and at what level of detail?

Figures (2)

• Figure 1: a. GRRAS $\kappa$-values and Dice Similarity Coefficient (DSC) scores define inter-observer agreement as "slight" to "almost perfect." Low agreement (red box) is unsuitable for medical applications. b. Explanation necessity is categorized based on: (i) variability in expert observations, and (ii) evaluation protocol robustness. Thresholds range from 0.00–0.10 (self-explainable) to 0.31–0.40 (new pattern discovery), varying across and within experience levels (Inexperienced, Experienced, Expert).
• Figure 2: a. The mathematical representation presents the overall XAI framework for a specific method $g$ across different explainability applications. b. Non-explanations involve methods $g(x,y,f)$ relying on inputs $x$, outputs $y$, and hidden parameters of $f(\theta,x)$. Significant annotation variance and discrepancies in ground truth extraction require global explanations $g_z$ using the entire dataset $D = {(x_i, y_i, z_i)}{i=1}^{n}$. c. In new-pattern discovery applications, there is the need to align global explanations $g_z$ with a ground truth statistical model $R(x')$ generated from a supergroup $D{x'}$. A transformation function $A_l(g_z,r_{x'})$ aligns the statistical shape model output with the global explanations. d. Proposed framework for explanation necessity, consisting of two flows: inter-observer variability assessment and dimensionality representation. Variability thresholds (Fig. \ref{['fc']}) classify 'Initial Explanation Necessity,' with adjudicating experts resolving discrepancies. Dimensionality evaluation refines the final 'Explanation Necessity Level' based on alignment or expert input.