Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a perturbation-based explanation for medical AI as differentiable programs

Takeshi Abe, Yoshiyuki Asai

TL;DR

The paper proposes perturbation-based explanations (PBX) for medical AI by leveraging the Jacobian $\mathbf{J}_f$ to measure sensitivity of outputs to input perturbations. It proves that the Jacobian of a deep network can be computed via a forward pass as $\mathbf{J}_f=\prod_{l=2}^L \frac{\partial a^{[l]}}{\partial a^{[l-1]}}$ when activation Jacobians are computable. A practical forward-pass algorithm (Algorithm 1) is described to obtain exact, instance-specific Jacobians without extra data, with the ability to extract intermediate Jacobians for partial models. Limitations include differentiability requirements and absence of higher-order interaction insights, guiding future work toward perturbation theory and extensions to other model families.

Abstract

Recent advancement in machine learning algorithms reaches a point where medical devices can be equipped with artificial intelligence (AI) models for diagnostic support and routine automation in clinical settings. In medicine and healthcare, there is a particular demand for sufficient and objective explainability of the outcome generated by AI models. However, AI models are generally considered as black boxes due to their complexity, and the computational process leading to their response is often opaque. Although several methods have been proposed to explain the behavior of models by evaluating the importance of each feature in discrimination and prediction, they may suffer from biases and opacities arising from the scale and sampling protocol of the dataset used for training or testing. To overcome the shortcomings of existing methods, we explore an alternative approach to provide an objective explanation of AI models that can be defined independently of the learning process and does not require additional data. As a preliminary study for this direction of research, this work examines a numerical availability of the Jacobian matrix of deep learning models that measures how stably a model responses against small perturbations added to the input. The indicator, if available, are calculated from a trained AI model for a given target input. This is a first step towards a perturbation-based explanation, which will assist medical practitioners in understanding and interpreting the response of the AI model in its clinical application.

Towards a perturbation-based explanation for medical AI as differentiable programs

TL;DR

The paper proposes perturbation-based explanations (PBX) for medical AI by leveraging the Jacobian to measure sensitivity of outputs to input perturbations. It proves that the Jacobian of a deep network can be computed via a forward pass as when activation Jacobians are computable. A practical forward-pass algorithm (Algorithm 1) is described to obtain exact, instance-specific Jacobians without extra data, with the ability to extract intermediate Jacobians for partial models. Limitations include differentiability requirements and absence of higher-order interaction insights, guiding future work toward perturbation theory and extensions to other model families.

Abstract

Recent advancement in machine learning algorithms reaches a point where medical devices can be equipped with artificial intelligence (AI) models for diagnostic support and routine automation in clinical settings. In medicine and healthcare, there is a particular demand for sufficient and objective explainability of the outcome generated by AI models. However, AI models are generally considered as black boxes due to their complexity, and the computational process leading to their response is often opaque. Although several methods have been proposed to explain the behavior of models by evaluating the importance of each feature in discrimination and prediction, they may suffer from biases and opacities arising from the scale and sampling protocol of the dataset used for training or testing. To overcome the shortcomings of existing methods, we explore an alternative approach to provide an objective explanation of AI models that can be defined independently of the learning process and does not require additional data. As a preliminary study for this direction of research, this work examines a numerical availability of the Jacobian matrix of deep learning models that measures how stably a model responses against small perturbations added to the input. The indicator, if available, are calculated from a trained AI model for a given target input. This is a first step towards a perturbation-based explanation, which will assist medical practitioners in understanding and interpreting the response of the AI model in its clinical application.

Paper Structure

This paper contains 5 sections, 1 theorem, 2 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Results
  4. Discussion
  5. Conclusion

Key Result

theorem thmcountertheorem

Let be $f$ a model represented in the above notation. if $\sigma^{\left[l\right]}$ has a Jacobian matrix that is computable at any point for each layer $l = 2, \dots, L$, then $f$'s Jacobian matrix $\mathbf{J}_f$ is computable at a given point.

Figures (1)

  • Figure 1: Schematic diagram for the standard model of deep learning.

Theorems & Definitions (2)

  • theorem thmcountertheorem
  • proof