When and How to Fool Explainable Models (and Humans) with Adversarial Examples

TL;DR

The paper addresses the vulnerability of explainable ML to adversarial manipulation by extending adversarial examples to scenarios where humans assess inputs, predictions, and explanations. It introduces a general framework that formalizes attack design across explanation types and user-centered scenarios, using notation such as $f(x)$, $h(x)$, $y_x$, $A_f(x)$, and $A_h(x)$, and analyzes eight S1–S8 scenarios. The contributions include formalizing extended adversarial definitions, outlining scenario-dependent attack requirements, and illustrating context-aware attacks on medical X-ray and ImageNet tasks with feature-based and prototype-based explanations. The work highlights practical implications for reliability and defense, offering a roadmap for rigorous evaluation and improved robustness of explainable AI systems.

Abstract

Reliable deployment of machine learning models such as neural networks continues to be challenging due to several limitations. Some of the main shortcomings are the lack of interpretability and the lack of robustness against adversarial examples or out-of-distribution inputs. In this exploratory review, we explore the possibilities and limits of adversarial attacks for explainable machine learning models. First, we extend the notion of adversarial examples to fit in explainable machine learning scenarios, in which the inputs, the output classifications and the explanations of the model's decisions are assessed by humans. Next, we propose a comprehensive framework to study whether (and how) adversarial examples can be generated for explainable models under human assessment, introducing and illustrating novel attack paradigms. In particular, our framework considers a wide range of relevant yet often ignored factors such as the type of problem, the user expertise or the objective of the explanations, in order to identify the attack strategies that should be adopted in each scenario to successfully deceive the model (and the human). The intention of these contributions is to serve as a basis for a more rigorous and realistic study of adversarial examples in the field of explainable machine learning.

Figures (8)

• Figure 1: Outline of our exploratory review.
• Figure 3: Illustration of an adversarial example generated for a chest X-ray (CXR) classification task, in which the objective is to categorize the status of the patient as one of three classes: "normal", Covid-19, or (non-Covid) pneumonia (more details in Section \ref{['sec:tasks_datasets_models']}). (a) Original input sample in which the patient is diagnosed with Covid-19. (b) Adversarially manipulated input, which is misclassified by the model as "normal" (i.e., no disease found in the patient) despite being perceptually identical to the original image.
• Figure 4: Attack casuistry when the human observes not only the input but also the output classification of the model.
• Figure 5: Critical scenarios to be considered in the study adversarial attacks against explainable machine learning models.
• Figure 6: Different types of adversarial attacks for the x-ray medical image diagnosis task. The left part of each image shows the input image as well as the class assigned by the model (jointly with the confidence score in the $[0,1]$ range), whereas the right part shows the explanation provided by the Grad-CAM method. (a) Original image. (b) Regular adversarial attack (PGD) targeting the class "normal" (i.e., the possible changes that the adversarial perturbation may produce in the explanation are not controlled by the attack). (c) Attack producing the wrong classification "normal" while maintaining the original explanation. (d) Attack maintaining the correct classification while changing the explanation in order to selectively highlight some parts (the right part) but omitting others (in this case, the left part). (e) Attack producing the wrong class "normal" and a wrong explanation which uniformly highlights the relevant parts of the image. (f) Attack producing the wrong class "normal" and a uniform explanation outside the main parts of the image (i.e., highlighting only irrelevant and incorrect parts).