Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation

TL;DR

The study assesses the robustness of medical vision-language segmentation models (VLSMs) to adversarial perturbations. It fine-tunes a CLIP-based segmentation model (CLIPSeg) with VL-Adapter blocks for medical anatomy segmentation and evaluates white-box attacks using FGSM and PGD across endoscopic, photographic, and radiographic datasets. Results show significant degradations in segmentation quality (DSC and IoU) under attack, with PGD generally more effective than FGSM and grayscale radiology data especially vulnerable; universal perturbations were not found. The work highlights a critical vulnerability of medical VLSMs and motivates development of defenses and broader attack-model evaluations.

Abstract

Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. \footnote{https://github.com/anjilab/secure-private-ai}

Figures (8)

• Figure 1: The overall methods of adversarial attack. $F_i$ and $F_t$ are fine-tuned image and text encoders, respectively. By fusing input images with the adversarial noise $\alpha$ generated from FGSM/PGD methods, we observe more inaccurate segmentation mask.
• Figure 2: Comparison of original images and adversarial images generated at different perturbation levels. As the perturbation increases, the adversarial modifications become increasingly perceptible to the human eye.
• Figure 3: Comparison of original images and adversarial images generated at different perturbation levels on Kvasir dataset using PGD. As the perturbation increases, the adversarial modifications become increasingly perceptible to the human eye.
• Figure 4: Comparison of original images and adversarial images generated at different perturbation levels on ISIC-16 dataset using FGSM. As the perturbation increases, the adversarial modifications become increasingly perceptible to the human eye.
• Figure 5: The provided image is a zoomed-in version highlighting the differences between the adversarial and original images. It offers a clearer comparison between the original and perturbed images generated via FGSM. This pattern is consistent across all other samples.