Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing for Complementarity: A Conceptual Framework to Go Beyond the Current Paradigm of Using XAI in Healthcare

Elisa Rubegni, Omran Ayoub, Stefania Maria Rita Rizzo, Marco Barbero, Guenda Bernegger, Francesca Faraci, Francesca Mangili, Emiliano Soldini, Pierpaolo Trimboli, Alessandro Facchini

TL;DR

This work addresses trust and adoption challenges of AI in healthcare by arguing that opacity and traditional explainability approaches can hinder effective human-AI collaboration. It proposes a human-centered, three-pillar XAI framework—Feature Importance, Counterexample Explanations, and Similar-Case Explanations—implemented within a participatory design process to support clinicians' diagnostic reasoning rather than issuing direct recommendations. The authors demonstrate the approach with a thyroid-disease diagnostic prototype built on a real dataset using an XGBoost classifier, with explanations provided via LIME and DiCE, and validated through iterative co-design workshops. The study offers a conceptual framework and a practical hi-fidelity prototype that emphasize complementarity between clinicians and AI, aiming to improve trust, validation, and virtuous adoption in clinical practice, with plans for broader evaluation.

Abstract

The widespread use of Artificial Intelligence-based tools in the healthcare sector raises many ethical and legal problems, one of the main reasons being their black-box nature and therefore the seemingly opacity and inscrutability of their characteristics and decision-making process. Literature extensively discusses how this can lead to phenomena of over-reliance and under-reliance, ultimately limiting the adoption of AI. We addressed these issues by building a theoretical framework based on three concepts: Feature Importance, Counterexample Explanations, and Similar-Case Explanations. Grounded in the literature, the model was deployed within a case study in which, using a participatory design approach, we designed and developed a high-fidelity prototype. Through the co-design and development of the prototype and the underlying model, we advanced the knowledge on how to design AI-based systems for enabling complementarity in the decision-making process in the healthcare domain. Our work aims at contributing to the current discourse on designing AI systems to support clinicians' decision-making processes.

Designing for Complementarity: A Conceptual Framework to Go Beyond the Current Paradigm of Using XAI in Healthcare

TL;DR

This work addresses trust and adoption challenges of AI in healthcare by arguing that opacity and traditional explainability approaches can hinder effective human-AI collaboration. It proposes a human-centered, three-pillar XAI framework—Feature Importance, Counterexample Explanations, and Similar-Case Explanations—implemented within a participatory design process to support clinicians' diagnostic reasoning rather than issuing direct recommendations. The authors demonstrate the approach with a thyroid-disease diagnostic prototype built on a real dataset using an XGBoost classifier, with explanations provided via LIME and DiCE, and validated through iterative co-design workshops. The study offers a conceptual framework and a practical hi-fidelity prototype that emphasize complementarity between clinicians and AI, aiming to improve trust, validation, and virtuous adoption in clinical practice, with plans for broader evaluation.

Abstract

The widespread use of Artificial Intelligence-based tools in the healthcare sector raises many ethical and legal problems, one of the main reasons being their black-box nature and therefore the seemingly opacity and inscrutability of their characteristics and decision-making process. Literature extensively discusses how this can lead to phenomena of over-reliance and under-reliance, ultimately limiting the adoption of AI. We addressed these issues by building a theoretical framework based on three concepts: Feature Importance, Counterexample Explanations, and Similar-Case Explanations. Grounded in the literature, the model was deployed within a case study in which, using a participatory design approach, we designed and developed a high-fidelity prototype. Through the co-design and development of the prototype and the underlying model, we advanced the knowledge on how to design AI-based systems for enabling complementarity in the decision-making process in the healthcare domain. Our work aims at contributing to the current discourse on designing AI systems to support clinicians' decision-making processes.
Paper Structure (15 sections, 1 figure, 1 table)

This paper contains 15 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. XAI: A Three Pillars Conceptual Framework
  4. Feature Importance
  5. Counterexample Explanations
  6. Similar-Case Example Explanations
  7. Designing with the Users: Methods, Data Collection and Analysis
  8. The Case Study
  9. Threats to Validity
  10. Phase 1: Understanding the Design Space
  11. Phase 2: Integrating the Theory into the Design
  12. ML Model Development and Explainers
  13. Prototype Interface and User Interaction
  14. Phase 3: Co-designing the Prototype
  15. Conclusion and Future Work

Figures (1)

  • Figure 1: Two screenshots of the prototype interface.