Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging

Poojitha Thota, Jai Prakash Veerla, Partha Sai Guttikonda, Mohammad S. Nasr, Shirin Nilizadeh, Jacob M. Luber

TL;DR

The paper investigates the vulnerability of a multimodal pathology Vision-Language Model (PLIP) to adversarial attacks. Using Projected Gradient Descent on 7,180 H&E patches from the Kather Colon dataset, the authors craft targeted perturbations that preserve perceptual similarity (SSIM) while forcing misclassifications. They report a 100% attack success rate across nine tissue types after 10 PGD steps and analyze changes in attention and prediction heatmaps to illuminate interpretability challenges. The work highlights concerns about the trustworthiness of pathology VLMs and discusses defense strategies such as adversarial training and diffusion-based purification. These results motivate robustness research to ensure reliable clinical deployment of multimodal models in medical imaging.

Abstract

In the context of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. Leveraging the Kather Colon dataset with 7,180 H&E images across nine tissue types, our investigation employs Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The outcomes reveal a 100% success rate in manipulating PLIP's predictions, underscoring its susceptibility to adversarial perturbations. The qualitative analysis of adversarial examples delves into the interpretability challenges, shedding light on nuanced changes in predictions induced by adversarial manipulations. These findings contribute crucial insights into the interpretability, domain adaptation, and trustworthiness of Vision Language Models in medical imaging. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models. The source codes for this experiment can be found at https://github.com/jaiprakash1824/VLM_Adv_Attack.

Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging

TL;DR

The paper investigates the vulnerability of a multimodal pathology Vision-Language Model (PLIP) to adversarial attacks. Using Projected Gradient Descent on 7,180 H&E patches from the Kather Colon dataset, the authors craft targeted perturbations that preserve perceptual similarity (SSIM) while forcing misclassifications. They report a 100% attack success rate across nine tissue types after 10 PGD steps and analyze changes in attention and prediction heatmaps to illuminate interpretability challenges. The work highlights concerns about the trustworthiness of pathology VLMs and discusses defense strategies such as adversarial training and diffusion-based purification. These results motivate robustness research to ensure reliable clinical deployment of multimodal models in medical imaging.

Abstract

In the context of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. Leveraging the Kather Colon dataset with 7,180 H&E images across nine tissue types, our investigation employs Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The outcomes reveal a 100% success rate in manipulating PLIP's predictions, underscoring its susceptibility to adversarial perturbations. The qualitative analysis of adversarial examples delves into the interpretability challenges, shedding light on nuanced changes in predictions induced by adversarial manipulations. These findings contribute crucial insights into the interpretability, domain adaptation, and trustworthiness of Vision Language Models in medical imaging. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models. The source codes for this experiment can be found at https://github.com/jaiprakash1824/VLM_Adv_Attack.
Paper Structure (14 sections, 6 figures)

This paper contains 14 sections, 6 figures.

Table of Contents

  1. Introduction
  2. methods
  3. Threat Model
  4. Vision Language Models - PLIP
  5. Dataset
  6. Adversarial Attack - Projected Gradient Descent
  7. Experiements and Results
  8. Evaluation
  9. Targeted Misclassification
  10. Discussions
  11. Possible Defenses
  12. Conclusion and Future Directions
  13. Acknowledgments
  14. Compliance with ethical standards

Figures (6)

  • Figure 1: Attack Overview
  • Figure 2: Original H&E + Pertubation = Adversarial H&E
  • Figure 3: PieChart of Kather Colon Dataset
  • Figure 4: Visualization of Attention before and after PGD attack on PLIP
  • Figure 5: Heatmaps showing distribution of Adversarial Attacks
  • ...and 1 more figures