Region-Guided Attack on the Segment Anything Model (SAM)

TL;DR

This work introduces Region-Guided Attack (RGA), a region-focused adversarial strategy against the Segment Anything Model (SAM). By constructing a Region-Guided Map (RGM) and applying a Segmentation and Dilation (SAD) approach, RGA targets large regions and expands small ones to distort segmentation with imperceptible perturbations. The method leverages 2D transformations and a feature-based loss to achieve high attack success and transferability across white-box and black-box settings, outperforming existing attacks. The findings reveal structural vulnerabilities in SAM's region-based segmentation and motivate development of robust defenses against region-guided perturbations for critical applications. Overall, RGA provides a principled, scalable framework for understanding and testing the resilience of advanced segmentation systems.

Abstract

The Segment Anything Model (SAM) is a cornerstone of image segmentation, demonstrating exceptional performance across various applications, particularly in autonomous driving and medical imaging, where precise segmentation is crucial. However, SAM is vulnerable to adversarial attacks that can significantly impair its functionality through minor input perturbations. Traditional techniques, such as FGSM and PGD, are often ineffective in segmentation tasks due to their reliance on global perturbations that overlook spatial nuances. Recent methods like Attack-SAM-K and UAD have begun to address these challenges, but they frequently depend on external cues and do not fully leverage the structural interdependencies within segmentation processes. This limitation underscores the need for a novel adversarial strategy that exploits the unique characteristics of segmentation tasks. In response, we introduce the Region-Guided Attack (RGA), designed specifically for SAM. RGA utilizes a Region-Guided Map (RGM) to manipulate segmented regions, enabling targeted perturbations that fragment large segments and expand smaller ones, resulting in erroneous outputs from SAM. Our experiments demonstrate that RGA achieves high success rates in both white-box and black-box scenarios, emphasizing the need for robust defenses against such sophisticated attacks. RGA not only reveals SAM's vulnerabilities but also lays the groundwork for developing more resilient defenses against adversarial threats in image segmentation.

Figures (8)

• Figure 1: Overview of the architecture of the Segment Anything Model (SAM), highlighting its key components and operational flow.
• Figure 2: Overview of the Region-Guided Attack (RGA) framework on SAM.
• Figure 3: Example of using SAD strategy to generate region-guided map.
• Figure 4: Qualitative comparison of segmentation results under different adversarial attacks on Meta AI’s online SAM and FastSAM models in a black-box setting. The comparison includes various prompts, with Point, Box, and Everything on Meta AI's online SAM, and Text ("dog") on FastSAM. The perturbation bound $\epsilon$ is reduced to $4/255$. While most other attack methods are largely ineffective, our method continues to successfully disrupt segmentation.
• Figure 5: Visualizations of generated adversarial perturbations from different methods, corresponding to different targets used during the attack on segmentation models.